[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 504948068 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2439 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22D5 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 002295 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2349 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23D9 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2411 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22d5-0xbffe2348] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22d4] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2349-0xbffe23d8] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23d9-0xbffe2410] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2411-0xbffe2438] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2560MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001013] APIC: Switch to symmetric I/O mode setup [ 0.003280] x2apic enabled [ 0.004013] Switched APIC routing to physical x2apic. [ 0.005018] kvm-guest: setup PV IPIs [ 0.008377] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009025] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010017] pid_max: default: 32768 minimum: 301 [ 0.012063] LSM: Security Framework initializing [ 0.013076] Yama: becoming mindful. [ 0.015039] SELinux: Initializing. [ 0.016066] *** VALIDATE selinux *** [ 0.026159] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.033763] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.034167] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.035142] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.037044] *** VALIDATE tmpfs *** [ 0.039019] *** VALIDATE proc *** [ 0.040000] *** VALIDATE cgroup *** [ 0.040011] *** VALIDATE cgroup2 *** [ 0.041274] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.042185] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.043009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.045023] Spectre V2 : User space: Vulnerable [ 0.046011] Speculative Store Bypass: Vulnerable [ 0.050395] debug: unmapping init [mem 0xffffffff88659000-0xffffffff88660fff] [ 0.053300] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.054820] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.055031] ... version: 2 [ 0.056015] ... bit width: 48 [ 0.057013] ... generic registers: 4 [ 0.058014] ... value mask: 0000ffffffffffff [ 0.059017] ... max period: 00007fffffffffff [ 0.060016] ... fixed-purpose events: 3 [ 0.061010] ... event mask: 000000070000000f [ 0.063249] rcu: Hierarchical SRCU implementation. [ 0.065706] smp: Bringing up secondary CPUs ... [ 0.066603] x86: Booting SMP configuration: [ 0.067016] .... node #0, CPUs: #1 #2 #3 [ 0.075399] smp: Brought up 1 node, 4 CPUs [ 0.076870] smpboot: Max logical packages: 1 [ 0.077010] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.108159] node 0 deferred pages initialised in 29ms [ 0.113185] devtmpfs: initialized [ 0.114438] x86/mm: Memory block size: 128MB [ 0.116383] gcov: version magic: 0x41383552 [ 0.118358] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.119077] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.120281] pinctrl core: initialized pinctrl subsystem [ 0.121201] [ 0.121841] ************************************************************* [ 0.122017] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.123014] ** ** [ 0.124023] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.125017] ** ** [ 0.126012] ** This means that this kernel is built to expose internal ** [ 0.127016] ** IOMMU data structures, which may compromise security on ** [ 0.128020] ** your system. ** [ 0.129020] ** ** [ 0.130027] ** If you see this message and you are not debugging the ** [ 0.131014] ** kernel, report this immediately to your vendor! ** [ 0.132022] ** ** [ 0.133019] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.134031] ************************************************************* [ 0.135934] NET: Registered protocol family 16 [ 0.136564] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.137081] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.138077] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.139766] cpuidle: using governor menu [ 0.140748] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.141726] PCI: Using configuration type 1 for base access [ 0.142257] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.149068] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.150010] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.152123] cryptd: max_cpu_qlen set to 1000 [ 0.153884] ACPI: Added _OSI(Module Device) [ 0.154000] ACPI: Added _OSI(Processor Device) [ 0.154024] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.156025] ACPI: Added _OSI(Processor Aggregator Device) [ 0.163021] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.172648] ACPI: Interpreter enabled [ 0.174090] ACPI: PM: (supports S0 S3 S4 S5) [ 0.175016] ACPI: Using IOAPIC for interrupt routing [ 0.177423] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.181777] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.196048] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.198048] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.202030] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.208097] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.217191] acpiphp: Slot [2] registered [ 0.219529] acpiphp: Slot [5] registered [ 0.223373] acpiphp: Slot [6] registered [ 0.225382] acpiphp: Slot [7] registered [ 0.230244] acpiphp: Slot [8] registered [ 0.232413] acpiphp: Slot [9] registered [ 0.234220] acpiphp: Slot [10] registered [ 0.235000] acpiphp: Slot [3] registered [ 0.238196] acpiphp: Slot [4] registered [ 0.240110] acpiphp: Slot [11] registered [ 0.244169] acpiphp: Slot [12] registered [ 0.245138] acpiphp: Slot [13] registered [ 0.247135] acpiphp: Slot [14] registered [ 0.249238] acpiphp: Slot [15] registered [ 0.251141] acpiphp: Slot [16] registered [ 0.253126] acpiphp: Slot [17] registered [ 0.255254] acpiphp: Slot [18] registered [ 0.257131] acpiphp: Slot [19] registered [ 0.259154] acpiphp: Slot [20] registered [ 0.262391] acpiphp: Slot [21] registered [ 0.264145] acpiphp: Slot [22] registered [ 0.266065] acpiphp: Slot [23] registered [ 0.267049] acpiphp: Slot [24] registered [ 0.268046] acpiphp: Slot [25] registered [ 0.269013] acpiphp: Slot [26] registered [ 0.270069] acpiphp: Slot [27] registered [ 0.271025] acpiphp: Slot [28] registered [ 0.272007] acpiphp: Slot [29] registered [ 0.274104] acpiphp: Slot [30] registered [ 0.275000] acpiphp: Slot [31] registered [ 0.275000] PCI host bridge to bus 0000:00 [ 0.276028] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.278064] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.281033] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.284057] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.287032] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.290110] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.293239] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.298223] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.302972] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.320024] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.327554] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.333027] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.338022] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.341022] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.345842] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.347795] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.353045] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.358007] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.365015] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.377019] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.384732] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.391000] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.400018] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.408020] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.422016] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.432000] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.441024] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.452022] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.467021] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.476000] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.482017] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.493018] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.511018] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.519888] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.527021] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.540023] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.570021] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.578112] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.584017] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.591028] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.605017] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.614267] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.618019] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.623020] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.634024] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.642396] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.644350] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.645308] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.647313] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.648186] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.652114] iommu: Default domain type: Passthrough [ 0.655438] SCSI subsystem initialized [ 0.656096] ACPI: bus type USB registered [ 0.657122] usbcore: registered new interface driver usbfs [ 0.659076] usbcore: registered new interface driver hub [ 0.661089] usbcore: registered new device driver usb [ 0.662176] pps_core: LinuxPPS API ver. 1 registered [ 0.664012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.666064] PTP clock support registered [ 0.668215] EDAC MC: Ver: 3.0.0 [ 0.670184] PCI: Using ACPI for IRQ routing [ 0.672038] NetLabel: Initializing [ 0.673008] NetLabel: domain hash size = 128 [ 0.674007] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.675118] NetLabel: unlabeled traffic allowed by default [ 0.677047] vgaarb: loaded [ 0.678432] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.680011] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.686319] clocksource: Switched to clocksource kvm-clock [ 0.812955] VFS: Disk quotas dquot_6.6.0 [ 0.814315] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.817278] *** VALIDATE ramfs *** [ 0.818532] *** VALIDATE hugetlbfs *** [ 0.819978] pnp: PnP ACPI init [ 0.822375] pnp: PnP ACPI: found 6 devices [ 0.839125] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.842216] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.844158] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.846316] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.848760] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.850930] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.853554] NET: Registered protocol family 2 [ 0.855836] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.860825] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.863523] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.868748] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.871908] TCP: Hash tables configured (established 65536 bind 65536) [ 0.874851] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.877722] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.880317] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.883611] NET: Registered protocol family 1 [ 0.885951] RPC: Registered named UNIX socket transport module. [ 0.888040] RPC: Registered udp transport module. [ 0.889570] RPC: Registered tcp transport module. [ 0.891205] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.893396] NET: Registered protocol family 44 [ 0.894901] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.896785] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.898757] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.900740] PCI: CLS 0 bytes, default 64 [ 0.902255] Unpacking initramfs... [ 3.078791] debug: unmapping init [mem 0xffff944dbcc54000-0xffff944dbffbffff] [ 3.094985] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 3.098448] software IO TLB: mapped [mem 0x00000000b8c54000-0x00000000bcc54000] (64MB) [ 3.102933] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.925529] Initialise system trusted keyrings [ 3.927487] Key type blacklist registered [ 3.929754] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.945158] zbud: loaded [ 3.951751] *** VALIDATE nfs *** [ 3.953518] *** VALIDATE nfs4 *** [ 3.955530] pstore: using deflate compression [ 3.960530] Platform Keyring initialized [ 4.121651] NET: Registered protocol family 38 [ 4.125736] Key type asymmetric registered [ 4.127340] Asymmetric key parser 'x509' registered [ 4.129829] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 4.133819] io scheduler mq-deadline registered [ 4.135923] io scheduler kyber registered [ 4.137682] io scheduler bfq registered [ 4.140549] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 4.143349] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 4.146463] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 4.149691] ACPI: Power Button [PWRF] [ 4.156290] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 4.164275] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 4.180122] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 4.189145] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 4.207855] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 4.237480] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 4.268432] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 4.274739] Non-volatile memory driver v1.3 [ 4.276545] Linux agpgart interface v0.103 [ 4.312079] virtio_blk virtio1: [vda] 134592 512-byte logical blocks (68.9 MB/65.7 MiB) [ 4.315094] vda: detected capacity change from 0 to 68911104 [ 4.337953] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.341015] vdb: detected capacity change from 0 to 1073741824 [ 4.359668] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.362317] vdc: detected capacity change from 0 to 2621440000 [ 4.380639] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.383523] vdd: detected capacity change from 0 to 2621440000 [ 4.401568] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.405448] vde: detected capacity change from 0 to 4294967296 [ 4.421678] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.427881] vdf: detected capacity change from 0 to 4294967296 [ 4.454629] libphy: Fixed MDIO Bus: probed [ 4.461710] usbcore: registered new interface driver usbserial_generic [ 4.463822] usbserial: USB Serial support registered for generic [ 4.465840] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.469673] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.471700] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.475081] mousedev: PS/2 mouse device common for all mice [ 4.480768] rtc_cmos 00:05: RTC can wake from S4 [ 4.483824] rtc_cmos 00:05: registered as rtc0 [ 4.486075] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.488019] intel_pstate: CPU model not supported [ 4.489337] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.501113] hid: raw HID events driver (C) Jiri Kosina [ 4.503818] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.506631] usbcore: registered new interface driver usbhid [ 4.512846] usbhid: USB HID core driver [ 4.514759] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.517808] drop_monitor: Initializing network drop monitor service [ 4.521507] Initializing XFRM netlink socket [ 4.523464] NET: Registered protocol family 10 [ 4.526962] Segment Routing with IPv6 [ 4.528507] NET: Registered protocol family 17 [ 4.530228] mpls_gso: MPLS GSO support [ 4.537047] RAS: Correctable Errors collector initialized. [ 4.538773] AVX version of gcm_enc/dec engaged. [ 4.540183] AES CTR mode by8 optimization enabled [ 4.630971] sched_clock: Marking stable (4630899135, 0)->(5679391597, -1048492462) [ 4.634645] registered taskstats version 1 [ 4.636598] Loading compiled-in X.509 certificates [ 4.638863] zswap: loaded using pool lzo/zbud [ 4.666789] Key type big_key registered [ 4.679965] Key type encrypted registered [ 4.681731] ima: No TPM chip found, activating TPM-bypass! [ 4.683626] ima: Allocated hash algorithm: sha1 [ 4.685545] ima: No architecture policies found [ 4.686916] evm: Initialising EVM extended attributes: [ 4.688722] evm: security.selinux [ 4.689844] evm: security.ima [ 4.691160] evm: security.capability [ 4.692572] evm: HMAC attrs: 0x1 [ 4.695336] rtc_cmos 00:05: setting system clock to 2026-03-11 12:38:19 UTC (1773232699) [ 4.703497] debug: unmapping init [mem 0xffffffff89603000-0xffffffff897fffff] [ 4.706510] debug: unmapping init [mem 0xffffffff88382000-0xffffffff88658fff] [ 4.710178] Write protecting the kernel read-only data: 28672k [ 4.715885] debug: unmapping init [mem 0xffffffff86a03000-0xffffffff86bfffff] [ 4.718456] debug: unmapping init [mem 0xffffffff87314000-0xffffffff873fffff] [ 4.794212] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.802729] systemd[1]: Detected virtualization kvm. [ 4.807562] systemd[1]: Detected architecture x86-64. [ 4.809459] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.838264] systemd[1]: No hostname configured. [ 4.840736] systemd[1]: Set hostname to . [ 4.843789] random: systemd: uninitialized urandom read (16 bytes read) [ 4.846878] systemd[1]: Initializing machine ID from random generator. [ 5.004755] random: ln: uninitialized urandom read (6 bytes read) [ 5.258985] random: systemd: uninitialized urandom read (16 bytes read) [ 5.263141] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 5.270569] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 5.276611] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Sockets. [ OK ] Reached target Timers. [ OK ] Reached target Local File Systems. [ OK ] Reached target Paths. Starting Apply Kernel Variables... [ OK ] Reached target Initrd Root Device. Starting Create Volatile Files and Directories... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Slices. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 6.600533] device-mapper: uevent: version 1.0.3 [ 6.603455] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 7.696759] virtio_net virtio0 ens2: renamed from eth0 [ 7.888683] random: fast init done [ 8.176938] scsi host0: ata_piix [ 8.232331] scsi host1: ata_piix [ 8.238551] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 8.241598] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 13.290869] random: crng init done [ 13.292715] random: 7 urandom warning(s) missed due to ratelimiting [ 15.597298] dracut-initqueue[585]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 17.547908] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 19.634575] printk: systemd: 26 output lines suppressed due to ratelimiting [ 20.302889] SELinux: Disabled at runtime. [ 20.368507] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 20.375571] systemd[1]: Detected virtualization kvm. [ 20.377218] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 21.548976] systemd[1]: initrd-switch-root.service: Succeeded. [ 21.553828] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 21.559391] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 21.567922] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 21.573503] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 21.585664] systemd[1]: Starting Journal Service... Starting Journal Service... [ 21.594842] systemd[1]: Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on initctl Compatibility Named Pipe. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on udev Control Socket. Starting Create list of required st…ce nodes for the current kernel... Mounting POSIX Message Queue File System... [ OK ] Listening on RPCbind Server Activation Socket. Starting udev Coldplug all Devices... [ OK ] Created slice User and Session Slice. [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-getty.slice. [ OK ] Reached target RPC Port Mapper. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. Mounting Huge Pages File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. Starting Remount Root and Kernel File Systems... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target Slices. Starting Apply Kernel Variables... Activating swap /dev/disk/by-label/SWAP... Mounting Kernel Debug File System... [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [ 22.096700] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 23.049654] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 24.664837] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 24.695445] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 27.582442] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 27.613585] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (7s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit)[ 30.127133] Key type dns_resolver registered [*** ] A start job is running for Configur…-only root support (8s / no limit)[ 30.586506] NFS: Registering the id_resolver key type [ 30.590231] Key type id_resolver registered [ 30.591802] Key type id_legacy registered [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Mark the need to relabel after reboot... [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Mark the need to relabel after reboot. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started D-Bus System Message Bus. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... [ OK ] Started GSSAPI Proxy Daemon. Starting Hostname Service... [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started OpenSSH server daemon. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. [ OK ] Started Authorization Manager. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg137-server login: [ 57.722683] libcfs: loading out-of-tree module taints kernel. [ 57.742803] Key type ._llcrypt registered [ 57.744218] Key type .llcrypt registered [ 57.791175] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_hostid [ 65.512108] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 66.145844] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 66.151676] alg: No test for adler32 (adler32-zlib) [ 67.157811] Lustre: Lustre: Build Version: 2.17.50_236_gaab9274 [ 67.471393] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 69.087237] Key type lgssc registered [ 69.869858] Lustre: Echo OBD driver; http://www.lustre.org/ [ 77.031243] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 93.263351] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 98.409769] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 98.419730] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 99.529139] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 99.543495] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 99.589590] Lustre: lustre-MDT0000: new disk, initializing [ 99.621156] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 99.630195] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 101.070430] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 106.844868] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 106.880287] Lustre: 6475:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 106.897106] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 106.899648] Lustre: Skipped 1 previous similar message [ 106.946628] Lustre: lustre-MDT0001: new disk, initializing [ 106.989909] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 107.010797] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 107.017248] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 108.479397] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 110.837147] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 114.304617] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 114.387752] Lustre: lustre-OST0000: new disk, initializing [ 114.389468] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 114.406837] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 116.299294] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 116.782740] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 116.786456] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 116.813147] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 121.404127] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 121.439112] Lustre: lustre-OST0001: new disk, initializing [ 121.441079] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 121.455572] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 123.096671] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 124.971321] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 124.976701] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 124.988489] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 128.863920] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 132.532086] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 139.604423] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing check_logdir /tmp/testlogs/ [ 141.216242] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing yml_node [ 142.853050] Lustre: DEBUG MARKER: Client: 2.17.50.236 [ 143.790773] Lustre: DEBUG MARKER: MDS: 2.17.50.236 [ 144.747995] Lustre: DEBUG MARKER: OSS: 2.17.50.236 [ 145.347651] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Wed Mar 11 08:40:39 EDT 2026 [ 151.232196] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 151.690609] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 163.295248] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 163.297414] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 163.301926] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 165.856179] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 165.856201] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 165.859028] Lustre: Skipped 2 previous similar messages [ 165.864277] Lustre: Skipped 2 previous similar messages [ 169.493610] Lustre: server umount lustre-MDT0000 complete [ 170.791627] LustreError: 6467:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773232865 with bad export cookie 7182404812055681743 [ 170.792911] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 170.795253] LustreError: 6467:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 170.975743] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 170.975961] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 170.983477] Lustre: Skipped 1 previous similar message [ 170.988079] Lustre: Skipped 2 previous similar messages [ 176.095951] LustreError: 6483:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 176.096086] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 176.100707] LustreError: 6483:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 4 previous similar messages [ 176.107334] Lustre: Skipped 1 previous similar message [ 181.215479] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 181.215484] LustreError: 6482:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 181.217573] Lustre: Skipped 1 previous similar message [ 181.222027] LustreError: 6482:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 185.311379] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 185.398596] Lustre: server umount lustre-MDT0001 complete [ 186.882349] Lustre: server umount lustre-OST0000 complete [ 188.446326] Lustre: server umount lustre-OST0001 complete [ 190.853958] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_hostid [ 194.191685] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 208.133124] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 208.245820] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 208.277409] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 208.325281] Lustre: lustre-MDT0000: new disk, initializing [ 208.373272] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 208.383186] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 209.963562] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 214.972125] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 215.023418] Lustre: 14939:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 215.151531] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 216.760800] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 220.185509] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 221.657321] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 224.538845] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 224.688941] Lustre: lustre-OST0000: new disk, initializing [ 224.691803] Lustre: Skipped 1 previous similar message [ 224.694892] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 224.696835] Lustre: Skipped 2 previous similar messages [ 224.734858] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 224.737263] Lustre: Skipped 1 previous similar message [ 225.729328] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 225.734079] Lustre: Skipped 1 previous similar message [ 225.737534] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 225.796309] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 227.196186] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 230.762249] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 232.528454] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 232.617744] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 234.226959] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 234.319935] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 236.002022] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 236.002245] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 236.004088] LustreError: Skipped 1 previous similar message [ 236.004800] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 236.008724] Lustre: Skipped 1 previous similar message [ 236.014013] Lustre: Skipped 1 previous similar message [ 241.243512] Lustre: server umount lustre-OST0000 complete [ 245.728156] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 245.731927] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 248.947890] Lustre: server umount lustre-MDT0000 complete [ 250.602722] LustreError: 14929:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773232945 with bad export cookie 7182404812055683262 [ 250.606880] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 250.608935] LustreError: 14929:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 250.768841] Lustre: server umount lustre-MDT0001 complete [ 252.486919] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 08:42:26 (1773232946) [ 253.511035] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 254.229663] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 08:42:28 (1773232948) [ 257.000988] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 257.204834] LustreError: 18938:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 257.232971] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 258.933497] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 262.307777] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 263.965354] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 265.999536] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 267.394637] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 269.940025] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 272.510581] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 274.717723] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 288.736437] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 288.740927] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 288.745664] Lustre: Skipped 1 previous similar message [ 288.748925] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 288.752230] Lustre: Skipped 5 previous similar messages [ 292.834608] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 294.360460] Lustre: server umount lustre-OST0000 complete [ 310.239113] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 310.314568] Lustre: server umount lustre-MDT0000 complete [ 311.927767] LustreError: 19657:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773233006 with bad export cookie 7182404812055684144 [ 311.931375] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 311.934606] LustreError: 19657:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 312.093553] Lustre: server umount lustre-MDT0001 complete [ 316.584576] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 317.813148] Key type lgssc unregistered [ 317.978651] LNet: 22138:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 317.983421] LNetError: 22138:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 317.993844] LNet: Removed LNI 192.168.201.137@tcp [ 318.459075] Key type .llcrypt unregistered [ 318.460903] Key type ._llcrypt unregistered [ 328.591392] Key type ._llcrypt registered [ 328.592942] Key type .llcrypt registered [ 328.640024] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 329.181759] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 329.206738] alg: No test for adler32 (adler32-zlib) [ 330.129523] Lustre: Lustre: Build Version: 2.17.50_236_gaab9274 [ 330.263439] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 331.871343] Key type lgssc registered [ 332.337564] Lustre: Echo OBD driver; http://www.lustre.org/ [ 335.515350] loop: module loaded [ 335.545827] loop0: detected capacity change from 0 to 204800000 [ 335.682954] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 335.688760] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 335.837702] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 338.390695] loop0: detected capacity change from 0 to 204800000 [ 338.470282] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 338.477790] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 338.484721] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 340.636949] loop0: detected capacity change from 0 to 204800000 [ 343.320446] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 343.328661] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 344.475430] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 344.505675] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 344.550302] Lustre: lustre-MDT0000: new disk, initializing [ 344.599239] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 344.614443] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 346.756381] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 351.734399] loop1: detected capacity change from 0 to 204800000 [ 354.626760] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 354.796304] Lustre: lustre-OST07c6: new disk, initializing [ 354.799315] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 354.802916] Lustre: Skipped 1 previous similar message [ 354.837841] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 358.030857] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 362.002987] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 362.008177] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 362.030159] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 367.138293] Lustre: 24273:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 367.144265] Lustre: 24273:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 367.148789] Lustre: 24273:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 367.154293] Lustre: 24273:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 367.161104] Lustre: 24273:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 367.167467] Lustre: 24273:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 388.892486] Lustre: 24272:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 388.896710] Lustre: 24272:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 388.899472] Lustre: 24272:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 388.902392] Lustre: 24272:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/4 [ 388.905413] Lustre: 24272:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 388.908909] Lustre: 24272:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 392.534322] Lustre: 24272:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 392.537154] Lustre: 24272:0:(osd_internal.h:1462:osd_trans_exec_op()) Skipped 1 previous similar message [ 392.538862] Lustre: 24272:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 392.541295] Lustre: 24272:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 392.544715] Lustre: 24272:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 392.546563] Lustre: 24272:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 392.549105] Lustre: 24272:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/23/0, punch: 0/0/0, quota 4/150/0 [ 392.551876] Lustre: 24272:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 392.554200] Lustre: 24272:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 392.556434] Lustre: 24272:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 392.558182] Lustre: 24272:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 392.560306] Lustre: 24272:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 395.620256] Lustre: 24274:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 395.624595] Lustre: 24274:0:(osd_internal.h:1462:osd_trans_exec_op()) Skipped 1 previous similar message [ 395.628446] Lustre: 24274:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 395.632329] Lustre: 24274:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 395.635765] Lustre: 24274:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 395.640287] Lustre: 24274:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 395.644024] Lustre: 24274:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/2 [ 395.648989] Lustre: 24274:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 395.652954] Lustre: 24274:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 395.656321] Lustre: 24274:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 395.660834] Lustre: 24274:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 395.665313] Lustre: 24274:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 402.296605] Lustre: 24273:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 402.302031] Lustre: 24273:0:(osd_internal.h:1462:osd_trans_exec_op()) Skipped 3 previous similar messages [ 402.305584] Lustre: 24273:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 402.309198] Lustre: 24273:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 402.312307] Lustre: 24273:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 402.315716] Lustre: 24273:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 402.319446] Lustre: 24273:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/2 [ 402.324510] Lustre: 24273:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 402.326940] Lustre: 24273:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 402.328983] Lustre: 24273:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 402.331976] Lustre: 24273:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 402.335070] Lustre: 24273:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 411.945199] Lustre: 24274:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 411.949164] Lustre: 24274:0:(osd_internal.h:1462:osd_trans_exec_op()) Skipped 5 previous similar messages [ 411.952323] Lustre: 24274:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 411.955688] Lustre: 24274:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 411.958072] Lustre: 24274:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 411.961295] Lustre: 24274:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 411.965016] Lustre: 24274:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/23/0, punch: 0/0/0, quota 4/150/0 [ 411.969284] Lustre: 24274:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 411.973361] Lustre: 24274:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 411.976312] Lustre: 24274:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 411.979820] Lustre: 24274:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 411.982531] Lustre: 24274:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 454.113573] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 454.113945] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 454.118927] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 459.232766] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 459.415696] Lustre: server umount lustre-OST07c6 complete [ 463.015581] Lustre: server umount lustre-MDT0000 complete [ 469.741156] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 471.065924] Key type lgssc unregistered [ 471.218867] LNet: 27813:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 471.223736] LNetError: 27813:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 471.234706] LNet: Removed LNI 192.168.201.137@tcp [ 471.618255] Key type .llcrypt unregistered [ 471.619870] Key type ._llcrypt unregistered [ 480.057467] Key type ._llcrypt registered [ 480.058764] Key type .llcrypt registered [ 480.111330] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 480.580956] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 480.639701] alg: No test for adler32 (adler32-zlib) [ 481.555433] Lustre: Lustre: Build Version: 2.17.50_236_gaab9274 [ 481.664100] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 483.263382] Key type lgssc registered [ 483.854559] Lustre: Echo OBD driver; http://www.lustre.org/ [ 487.349529] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 487.360716] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 488.581746] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 490.460168] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 494.742248] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 494.972886] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 496.773434] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 499.181892] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 500.799990] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 503.613977] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 503.765482] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 506.052766] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 508.286853] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 514.026414] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 525.279958] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 525.284912] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 525.292694] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 526.819028] LustreError: lustre-OST0000-osc-MDT0000: operation ost_setattr to node 0@lo failed: rc = -107 [ 526.819154] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 526.821843] LustreError: Skipped 1 previous similar message [ 526.827800] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 529.377710] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 529.380539] Lustre: Skipped 1 previous similar message [ 530.780177] Lustre: server umount lustre-OST0000 complete [ 534.496903] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 534.502082] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 534.504393] Lustre: Skipped 1 previous similar message [ 538.516926] Lustre: server umount lustre-MDT0000 complete [ 539.617798] LustreError: 30170:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 539.623666] LustreError: 30170:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 3 previous similar messages [ 539.834200] LustreError: 29027:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773233234 with bad export cookie 1589784810531066484 [ 539.837927] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 539.838670] LustreError: 29027:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 539.973966] Lustre: server umount lustre-MDT0001 complete [ 543.880140] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 545.021179] Key type lgssc unregistered [ 545.168744] LNet: 32443:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 545.171443] LNetError: 32443:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 545.180326] LNet: Removed LNI 192.168.201.137@tcp [ 545.539105] Key type .llcrypt unregistered [ 545.540506] Key type ._llcrypt unregistered [ 553.461903] Key type ._llcrypt registered [ 553.463324] Key type .llcrypt registered [ 553.509911] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 554.086565] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 554.097255] alg: No test for adler32 (adler32-zlib) [ 554.976017] Lustre: Lustre: Build Version: 2.17.50_236_gaab9274 [ 555.081348] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 556.671145] Key type lgssc registered [ 557.119069] Lustre: Echo OBD driver; http://www.lustre.org/ [ 560.451948] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 560.461328] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 561.684219] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 563.287213] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 566.535436] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 566.709345] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 568.364865] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 570.417427] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 571.824369] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 574.354499] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 574.514996] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 576.771870] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 578.942347] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 586.800145] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 596.961272] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 596.962825] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 596.964833] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 596.969713] Lustre: Skipped 1 previous similar message [ 600.101590] Lustre: server umount lustre-OST0000 complete [ 602.081049] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 602.081758] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 602.088042] Lustre: Skipped 1 previous similar message [ 602.095070] Lustre: Skipped 2 previous similar messages [ 607.201397] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 607.206147] Lustre: Skipped 1 previous similar message [ 607.865595] Lustre: server umount lustre-MDT0000 complete [ 609.493198] LustreError: 35287:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773233304 with bad export cookie 4317837208234911535 [ 609.496637] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 609.497277] LustreError: 35287:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 609.635662] Lustre: server umount lustre-MDT0001 complete [ 614.181386] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 615.656874] Key type lgssc unregistered [ 615.812955] LNet: 36872:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 615.817582] LNetError: 36872:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 615.826861] LNet: Removed LNI 192.168.201.137@tcp [ 616.236464] Key type .llcrypt unregistered [ 616.238482] Key type ._llcrypt unregistered [ 627.179334] Key type ._llcrypt registered [ 627.180826] Key type .llcrypt registered [ 627.224456] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 627.613788] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 627.666065] alg: No test for adler32 (adler32-zlib) [ 628.566457] Lustre: Lustre: Build Version: 2.17.50_236_gaab9274 [ 628.666829] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 630.255157] Key type lgssc registered [ 630.696094] Lustre: Echo OBD driver; http://www.lustre.org/ [ 633.769327] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 633.778910] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 634.974107] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 636.549173] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 639.826906] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 640.004717] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 641.744564] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 644.116595] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 645.573812] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 648.495556] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 648.695084] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 651.181600] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 653.394925] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 660.968038] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 669.104592] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.37@tcp (stopping) [ 670.687846] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 670.690375] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 670.694550] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 671.203857] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 671.210658] Lustre: Skipped 1 previous similar message [ 672.806634] Lustre: server umount lustre-MDT0000 complete [ 674.181254] LustreError: 38227:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773233368 with bad export cookie 9275751756999013576 [ 674.185631] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 674.185872] LustreError: 38227:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 674.212538] LustreError: 40401:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 674.218101] LustreError: 40401:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 674.221618] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.37@tcp (stopping) [ 674.224505] Lustre: Skipped 3 previous similar messages [ 676.320119] LustreError: 38241:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 676.320793] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 676.331546] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 679.327672] LustreError: 38243:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 679.336207] LustreError: 38243:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 680.381761] Lustre: server umount lustre-MDT0001 complete [ 698.394970] Lustre: server umount lustre-OST0000 complete [ 703.772536] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 704.829219] Key type lgssc unregistered [ 704.962433] LNet: 41776:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 704.964794] LNetError: 41776:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 704.970472] LNet: Removed LNI 192.168.201.137@tcp [ 705.275671] Key type .llcrypt unregistered [ 705.277306] Key type ._llcrypt unregistered [ 715.345260] Key type ._llcrypt registered [ 715.346651] Key type .llcrypt registered [ 715.396489] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 715.828433] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 715.889786] alg: No test for adler32 (adler32-zlib) [ 716.757482] Lustre: Lustre: Build Version: 2.17.50_236_gaab9274 [ 716.863966] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 718.455135] Key type lgssc registered [ 718.889983] Lustre: Echo OBD driver; http://www.lustre.org/ [ 721.932324] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 721.940076] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 723.125502] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 724.487356] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 727.696691] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 727.840978] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 729.402782] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 731.414595] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 732.919374] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 735.710281] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 735.862552] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 738.068334] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 740.211563] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 748.136581] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 753.633632] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 753.635322] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 753.636996] LustreError: Skipped 1 previous similar message [ 753.640634] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 753.643393] Lustre: Skipped 1 previous similar message [ 756.143258] Lustre: lustre-OST0000: Not available for connect from 192.168.201.37@tcp (stopping) [ 758.756170] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 758.759252] Lustre: Skipped 1 previous similar message [ 759.320093] Lustre: server umount lustre-OST0000 complete [ 772.506934] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 772.509982] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 772.512137] Lustre: Skipped 2 previous similar messages [ 772.519590] Lustre: Skipped 1 previous similar message [ 777.696178] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 777.700583] Lustre: Skipped 1 previous similar message [ 778.780991] Lustre: server umount lustre-MDT0000 complete [ 780.251319] LustreError: 45416:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773233475 with bad export cookie 17820230568153197883 [ 780.255147] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 780.256245] LustreError: 45416:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 786.431406] Lustre: server umount lustre-MDT0001 complete [ 790.514924] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 791.684933] Key type lgssc unregistered [ 791.833581] LNet: 46528:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 791.836715] LNetError: 46528:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 791.846356] LNet: Removed LNI 192.168.201.137@tcp [ 792.167951] Key type .llcrypt unregistered [ 792.168870] Key type ._llcrypt unregistered [ 802.206984] Key type ._llcrypt registered [ 802.208343] Key type .llcrypt registered [ 802.251263] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 802.684413] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 802.743212] alg: No test for adler32 (adler32-zlib) [ 803.607616] Lustre: Lustre: Build Version: 2.17.50_236_gaab9274 [ 803.706995] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 805.295151] Key type lgssc registered [ 805.696172] Lustre: Echo OBD driver; http://www.lustre.org/ [ 808.468417] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 808.474687] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 809.634279] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 810.961107] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 814.260071] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 814.416902] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 815.977676] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 818.125622] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 819.565204] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 821.929275] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 822.036323] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 823.959382] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 825.840439] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 834.342500] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 838.731958] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 840.030688] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 850.707344] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-03-11 8h52m24s [ 851.163313] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 851.882949] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.37@tcp (stopping) [ 852.900457] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.37@tcp (stopping) [ 855.009376] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 855.011189] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 855.014209] Lustre: Skipped 2 previous similar messages [ 855.016384] Lustre: Skipped 1 previous similar message [ 859.044653] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.37@tcp (stopping) [ 859.048621] Lustre: Skipped 3 previous similar messages [ 864.100298] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.37@tcp (stopping) [ 864.103325] Lustre: Skipped 4 previous similar messages [ 866.271394] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 866.339334] Lustre: server umount lustre-MDT0000 complete [ 867.623158] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 868.320595] LustreError: 47904:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 868.327802] LustreError: 47904:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 4 previous similar messages [ 870.066430] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 870.115994] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 870.228859] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 870.256849] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 871.273979] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 871.839657] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 873.100177] Lustre: DEBUG MARKER: Wait for df (35987) ... [ 873.650485] Lustre: DEBUG MARKER: done [ 875.488613] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 875.495189] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 875.500528] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 876.514333] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 876.515032] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 876.884714] Lustre: server umount lustre-OST0000 complete [ 880.406038] Lustre: server umount lustre-MDT0000 complete [ 881.632555] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 881.632934] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 881.633514] LustreError: 49141:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 881.642227] Lustre: Skipped 1 previous similar message [ 881.655567] LustreError: 47884:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773233576 with bad export cookie 4185419205047937952 [ 881.659054] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 881.659534] LustreError: 47884:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 887.874690] Lustre: server umount lustre-MDT0001 complete [ 891.829747] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 893.071540] Key type lgssc unregistered [ 893.226395] LNet: 52481:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 893.229559] LNetError: 52481:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 893.238278] LNet: Removed LNI 192.168.201.137@tcp [ 893.577166] Key type .llcrypt unregistered [ 893.580106] Key type ._llcrypt unregistered [ 909.990463] Key type ._llcrypt registered [ 909.992132] Key type .llcrypt registered [ 910.036673] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 910.539450] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 910.583595] alg: No test for adler32 (adler32-zlib) [ 911.484910] Lustre: Lustre: Build Version: 2.17.50_236_gaab9274 [ 911.620218] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 913.231215] Key type lgssc registered [ 913.653076] Lustre: Echo OBD driver; http://www.lustre.org/ [ 916.467107] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 916.472754] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 917.579065] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 917.592109] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 917.665745] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 919.048651] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 922.304028] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 922.347931] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 922.361764] Lustre: 54651:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 922.470679] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 923.872310] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 925.831838] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 927.216730] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 929.725162] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 929.800723] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 929.864677] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 930.645694] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 931.930202] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 933.957445] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 937.489793] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-03-11 8h53m51s [ 938.096131] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 940.481636] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 941.351623] Lustre: DEBUG MARKER: Creating a test file and stat it [ 941.381763] LustreError: 54659:0:(mdt_handler.c:2334:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 961.460137] Lustre: lustre-MDT0000: Client 6f78b091-64c1-45c4-bc56-ad0b6c1595ae (at 192.168.201.37@tcp) reconnecting [ 981.431065] LustreError: 54659:0:(mdt_handler.c:2334:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 982.406286] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 983.296981] Lustre: DEBUG MARKER: done [ 987.105086] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 987.108634] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 987.110249] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 987.111113] LustreError: Skipped 1 previous similar message [ 987.116359] Lustre: Skipped 1 previous similar message [ 991.187088] Lustre: server umount lustre-OST0000 complete [ 994.271968] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 994.274080] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 994.277286] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 994.278910] Lustre: Skipped 1 previous similar message [ 995.296116] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 995.298276] Lustre: Skipped 2 previous similar messages [ 998.370295] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 998.372782] Lustre: Skipped 1 previous similar message [ 998.550954] Lustre: server umount lustre-MDT0000 complete [ 999.908779] LustreError: 54643:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773233694 with bad export cookie 3610514769051121057 [ 999.913622] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 999.916019] LustreError: 54643:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1000.054244] Lustre: server umount lustre-MDT0001 complete [ 1003.854272] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 1004.894653] Key type lgssc unregistered [ 1005.018316] LNet: 58618:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1005.020193] LNetError: 58618:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1005.030290] LNet: Removed LNI 192.168.201.137@tcp [ 1005.297735] Key type .llcrypt unregistered [ 1005.298760] Key type ._llcrypt unregistered [ 1021.213697] Key type ._llcrypt registered [ 1021.215039] Key type .llcrypt registered [ 1021.256821] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 1021.652707] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1021.698816] alg: No test for adler32 (adler32-zlib) [ 1022.573299] Lustre: Lustre: Build Version: 2.17.50_236_gaab9274 [ 1022.687973] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 1024.287147] Key type lgssc registered [ 1024.703762] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1025.986827] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1025.995183] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1027.114757] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1027.128844] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1027.203033] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1027.588123] Lustre: Failing over lustre-MDT0000 [ 1027.697893] Lustre: server umount lustre-MDT0000 complete [ 1029.898160] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 08:55:23 (1773233723) [ 1032.056859] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1032.208986] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1033.561065] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1036.430814] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1036.469660] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1036.486084] Lustre: 61378:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1036.599114] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1037.931761] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1039.697674] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1041.002125] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1043.175163] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1043.261152] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1043.329375] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1044.129442] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1045.346122] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1047.350103] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1050.509376] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1054.689857] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1054.690744] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1054.693074] Lustre: Skipped 2 previous similar messages [ 1054.695610] Lustre: Skipped 1 previous similar message [ 1058.393152] Lustre: server umount lustre-MDT0000 complete [ 1059.639968] LustreError: 61368:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773233754 with bad export cookie 13857633050773063117 [ 1059.642801] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1059.642803] LustreError: 61368:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1059.753828] Lustre: server umount lustre-MDT0001 complete [ 1060.892539] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1062.347226] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1065.154913] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1070.047667] LustreError: 60299:0:(client.c:1428:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000009d991f7a x1859370318068352/t0(0) o250->MGC192.168.201.137@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1070.216874] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1070.251774] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1071.664029] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1074.340614] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1074.340944] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1074.793299] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1075.343077] Lustre: 60301:0:(client.c:2516:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773233754/real 1773233754] req@00000000c78913ca x1859370318067968/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1773233770 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1076.552190] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1078.692992] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1080.239215] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1080.294153] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1081.311164] Lustre: 60303:0:(client.c:2516:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773233759/real 1773233759] req@0000000041585be9 x1859370318068096/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1773233775 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1085.410376] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1085.411369] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1085.412037] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1085.416986] Lustre: Skipped 1 previous similar message [ 1086.431145] Lustre: 60303:0:(client.c:2516:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773233764/real 1773233764] req@000000006dce64a1 x1859370318068224/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1773233780 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1088.220454] Lustre: server umount lustre-MDT0000 complete [ 1089.520986] LustreError: 63013:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773233784 with bad export cookie 13857633050773065385 [ 1089.522865] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1089.523826] LustreError: 63013:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1089.646803] Lustre: server umount lustre-MDT0001 complete [ 1092.237639] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1095.752205] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1099.680089] LustreError: 60299:0:(client.c:1428:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000002115bc5c x1859370318086912/t0(0) o250->MGC192.168.201.137@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1099.808536] LustreError: 62997:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1099.822034] LustreError: 62997:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 4 previous similar messages [ 1099.841635] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1099.844842] Lustre: Skipped 1 previous similar message [ 1099.867866] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 1101.478218] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1104.865507] LustreError: 62996:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1104.866061] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1104.876148] Lustre: Skipped 1 previous similar message [ 1104.915708] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1106.149117] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1106.895359] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1107.168387] Lustre: 60301:0:(client.c:2516:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773233785/real 1773233785] req@00000000e56d7f92 x1859370318086528/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1773233801 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1109.186471] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1110.726164] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1111.263117] Lustre: 60302:0:(client.c:2516:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773233790/real 1773233790] req@00000000d322c961 x1859370318086784/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1773233806 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1111.272512] Lustre: 60302:0:(client.c:2516:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1115.616517] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1115.618962] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1115.624144] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1115.626239] Lustre: Skipped 2 previous similar messages [ 1118.945512] Lustre: server umount lustre-MDT0000 complete [ 1120.311302] LustreError: 61368:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773233815 with bad export cookie 13857633050773066764 [ 1120.314194] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1120.317460] LustreError: 61368:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1120.443580] Lustre: server umount lustre-MDT0001 complete [ 1122.845084] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1124.087034] Lustre: DEBUG MARKER: files compared the same [ 1135.137607] Lustre: server umount lustre-OST0000 complete [ 1140.459710] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 1141.456578] Key type lgssc unregistered [ 1141.576320] LNet: 70292:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1141.578198] LNetError: 70292:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1141.584291] LNet: Removed LNI 192.168.201.137@tcp [ 1141.875429] Key type .llcrypt unregistered [ 1141.876734] Key type ._llcrypt unregistered [ 1150.812781] Key type ._llcrypt registered [ 1150.813787] Key type .llcrypt registered [ 1150.857119] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 1151.279271] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1151.315112] alg: No test for adler32 (adler32-zlib) [ 1152.187160] Lustre: Lustre: Build Version: 2.17.50_236_gaab9274 [ 1152.286355] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 1153.871145] Key type lgssc registered [ 1154.257657] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1156.929769] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1156.936269] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1158.109676] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1159.320019] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug +malloc all [ 1162.065257] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1162.172650] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1163.525118] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug +malloc all [ 1165.337835] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1166.608747] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1168.800213] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1168.895707] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1170.803539] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug +malloc all [ 1172.626809] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1182.242346] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 1187.297428] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1187.297492] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1187.297526] LustreError: Skipped 1 previous similar message [ 1187.298022] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1187.299934] Lustre: Skipped 1 previous similar message [ 1191.699141] Lustre: server umount lustre-OST0000 complete [ 1197.535887] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1197.536189] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1197.539387] Lustre: Skipped 1 previous similar message [ 1197.541053] Lustre: Skipped 1 previous similar message [ 1202.655687] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1202.658109] Lustre: Skipped 2 previous similar messages [ 1207.263177] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1207.314976] Lustre: server umount lustre-MDT0000 complete [ 1207.775568] LustreError: 71668:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1207.780093] LustreError: 71668:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 3 previous similar messages [ 1208.558735] LustreError: 73830:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773233903 with bad export cookie 4607110610518611930 [ 1208.560528] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1208.563584] LustreError: 73830:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1208.686317] Lustre: server umount lustre-MDT0001 complete [ 1212.122649] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 1213.069084] Key type lgssc unregistered [ 1213.191336] LNet: 74867:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1213.193954] LNetError: 74867:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1213.206294] LNet: Removed LNI 192.168.201.137@tcp [ 1213.500653] Key type .llcrypt unregistered [ 1213.501912] Key type ._llcrypt unregistered [ 1222.095238] Key type ._llcrypt registered [ 1222.096984] Key type .llcrypt registered [ 1222.140040] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 1222.686044] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1222.694957] alg: No test for adler32 (adler32-zlib) [ 1223.544262] Lustre: Lustre: Build Version: 2.17.50_236_gaab9274 [ 1223.620795] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 1225.207211] Key type lgssc registered [ 1225.560521] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1228.086706] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1228.092987] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1243.551220] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1248.671297] LustreError: 76216:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.137@tcp: failed processing log, type 4: rc = -110 [ 1275.295255] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1277.324793] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1279.060037] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1281.490194] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1281.532167] Lustre: *** cfs_fail_loc=706, val=0*** [ 1281.673137] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1281.692754] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 1282.875459] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1285.749343] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1285.870082] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1287.082723] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1288.754344] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1289.962634] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1291.232326] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1291.234363] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1291.237780] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1292.256029] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1292.258711] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1292.263097] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1292.747274] Lustre: server umount lustre-OST0000 complete [ 1295.329087] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1295.329951] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1295.334234] Lustre: Skipped 1 previous similar message [ 1295.337336] Lustre: Skipped 1 previous similar message [ 1300.061301] Lustre: server umount lustre-MDT0000 complete [ 1300.448053] LustreError: 77191:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1300.452425] LustreError: 77191:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 3 previous similar messages [ 1301.250246] LustreError: 76222:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773233996 with bad export cookie 8466475648867360393 [ 1301.253239] LustreError: 76222:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1301.253540] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1301.371971] Lustre: server umount lustre-MDT0001 complete [ 1305.171930] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 1306.134482] Key type lgssc unregistered [ 1306.250361] LNet: 79478:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1306.252552] LNetError: 79478:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1306.264268] LNet: Removed LNI 192.168.201.137@tcp [ 1306.534263] Key type .llcrypt unregistered [ 1306.535371] Key type ._llcrypt unregistered [ 1315.286102] Key type ._llcrypt registered [ 1315.287435] Key type .llcrypt registered [ 1315.330623] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 1315.639665] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1315.647322] alg: No test for adler32 (adler32-zlib) [ 1316.498262] Lustre: Lustre: Build Version: 2.17.50_236_gaab9274 [ 1316.580965] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 1318.167102] Key type lgssc registered [ 1318.539981] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1321.070046] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1321.075897] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1323.357712] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1325.878527] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1326.059501] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1327.254289] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1330.084754] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1330.202816] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1332.156222] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1334.913259] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1334.917644] Lustre: MGS: Client 91d28d86-e41b-4a5d-80ca-72f2c25e7575 (at 0@lo) reconnecting [ 1334.924697] LustreError: 81536:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 1335.013627] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1335.050579] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 1336.295691] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1345.504290] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1345.506808] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1345.511859] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1346.530914] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1346.531293] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1346.533865] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1350.093198] Lustre: server umount lustre-OST0000 complete [ 1351.648424] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1351.649048] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1351.649372] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1351.649376] Lustre: Skipped 1 previous similar message [ 1351.653515] Lustre: Skipped 1 previous similar message [ 1356.768212] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1356.771307] Lustre: Skipped 2 previous similar messages [ 1357.389770] Lustre: server umount lustre-MDT0000 complete [ 1358.784937] Lustre: server umount lustre-MDT0001 complete [ 1360.066263] Lustre: server umount MGS complete [ 1363.474303] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 09:00:57 (1773234057) [ 1372.059617] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_hostid [ 1374.740576] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 1385.857047] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1387.099896] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1390.779656] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1390.861207] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 1390.877453] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1390.911887] Lustre: lustre-MDT0001: new disk, initializing [ 1390.934607] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1390.942888] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1390.944980] Lustre: 88592:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 1390.947555] LustreError: 88592:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 1392.172882] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1396.139464] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1396.233145] Lustre: lustre-OST0000: new disk, initializing [ 1396.235223] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1396.255350] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1398.062308] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1401.126739] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1402.399836] Lustre: 89944:0:(client.c:1675:after_reply()) @@@ resending request on EINPROGRESS req@00000000df326abb x1859370626352896/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1773234113 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 1403.484812] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1403.489241] Lustre: MGS: Client 8286573d-9ca1-4b2a-a4c7-56abd1108780 (at 0@lo) reconnecting [ 1403.503802] Lustre: 90272:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1403.513607] LustreError: 88578:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 1403.516531] LustreError: 88578:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 1403.522641] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1403.560489] Lustre: lustre-MDT0000: new disk, initializing [ 1403.626909] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1404.640698] Lustre: 90299:0:(client.c:1675:after_reply()) @@@ resending request on EINPROGRESS req@0000000032b16ab3 x1859370626358400/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1773234115 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 1404.703601] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1404.795403] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1407.457910] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 1407.460847] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 1409.698024] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 1409.701375] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 1409.763138] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000401 [ 1430.496075] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1430.499048] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1430.504420] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1434.573857] Lustre: server umount lustre-OST0000 complete [ 1438.176847] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1438.177398] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1438.179152] LustreError: Skipped 1 previous similar message [ 1438.182273] Lustre: Skipped 2 previous similar messages [ 1442.052299] Lustre: server umount lustre-MDT0000 complete [ 1443.297192] LustreError: 90722:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1443.304097] LustreError: 90722:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 6 previous similar messages [ 1449.519285] Lustre: server umount lustre-MDT0001 complete [ 1450.814353] Lustre: server umount MGS complete [ 1454.438811] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 09:02:28 (1773234148) [ 1459.911463] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 1460.898874] Key type lgssc unregistered [ 1461.031346] LNet: 92762:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1461.033763] LNetError: 92762:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1461.040487] LNet: Removed LNI 192.168.201.137@tcp [ 1461.333051] Key type .llcrypt unregistered [ 1461.334078] Key type ._llcrypt unregistered [ 1468.100769] Key type ._llcrypt registered [ 1468.101774] Key type .llcrypt registered [ 1468.137754] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 1468.553971] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1468.560246] alg: No test for adler32 (adler32-zlib) [ 1469.432560] Lustre: Lustre: Build Version: 2.17.50_236_gaab9274 [ 1469.528693] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 1471.119223] Key type lgssc registered [ 1471.561179] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1474.393151] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1474.399536] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1475.591858] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1476.950410] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1479.919466] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1480.048493] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1481.407850] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1483.092594] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1484.247992] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1486.448822] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1486.560482] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1488.304640] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1489.981141] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1501.926499] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:33) [ 1510.367880] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1510.370282] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1510.374854] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1512.417198] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1512.418425] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1512.419506] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1512.420388] Lustre: Skipped 1 previous similar message [ 1514.319349] Lustre: server umount lustre-OST0000 complete [ 1516.000168] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1516.002506] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1516.006304] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1521.685309] Lustre: server umount lustre-MDT0000 complete [ 1522.656148] LustreError: 93944:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1522.661463] LustreError: 93944:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 3 previous similar messages [ 1522.918264] LustreError: 93970:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773234217 with bad export cookie 4235839258040636837 [ 1522.922424] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1522.923039] LustreError: 93970:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1523.032977] Lustre: server umount lustre-MDT0001 complete [ 1526.559438] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 1527.510841] Key type lgssc unregistered [ 1527.626291] LNet: 97143:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1527.628835] LNetError: 97143:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1527.635309] LNet: Removed LNI 192.168.201.137@tcp [ 1527.917145] Key type .llcrypt unregistered [ 1527.918723] Key type ._llcrypt unregistered [ 1534.215261] Key type ._llcrypt registered [ 1534.217044] Key type .llcrypt registered [ 1534.260523] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 1534.600236] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1534.630098] alg: No test for adler32 (adler32-zlib) [ 1535.481125] Lustre: Lustre: Build Version: 2.17.50_236_gaab9274 [ 1535.567340] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 1537.143169] Key type lgssc registered [ 1537.512890] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1539.576316] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1539.578378] LustreError: 98187:0:(tgt_mount.c:2522:server_fill_super()) cfs_race id 716 sleeping [ 1539.579883] Lustre: Skipped 1 previous similar message [ 1539.581904] LustreError: 98186:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 waking [ 1539.583980] LustreError: 98187:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 awake: rc=4995 [ 1539.587307] LustreError: 98187:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 1539.589936] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1539.590233] LustreError: 98187:0:(super25.c:186:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1540.756911] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1543.373566] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1543.484953] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1544.895205] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1546.505430] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing lsmod [ 1547.539129] LustreError: 99359:0:(tgt_mount.c:2522:server_fill_super()) cfs_race id 716 sleeping [ 1547.542811] LustreError: 99360:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 waking [ 1547.546200] LustreError: 99359:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 awake: rc=4995 [ 1547.549034] LustreError: 99359:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 1547.552980] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1547.553731] LustreError: 99359:0:(super25.c:186:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1547.684566] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1548.615250] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1548.616071] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1548.618815] Lustre: Skipped 1 previous similar message [ 1553.761953] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1553.763654] Lustre: Skipped 4 previous similar messages [ 1554.774751] Lustre: server umount lustre-MDT0000 complete [ 1555.974150] LustreError: 98201:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773234250 with bad export cookie 1573782762209291209 [ 1555.977774] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1555.978995] LustreError: 98201:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1557.855239] LustreError: 98657:0:(client.c:1418:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000036570254 x1859370855971584/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1557.902422] Lustre: server umount lustre-MDT0001 complete [ 1568.351301] Lustre: server umount lustre-OST0000 complete [ 1571.102688] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1571.251411] LustreError: 100424:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1571.258400] LustreError: 100424:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 1571.278164] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1572.570420] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1575.323704] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1575.465780] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1576.737344] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1578.479337] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1579.601663] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1581.731973] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1583.620177] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1585.375330] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1597.222294] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:65) [ 1602.529910] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1602.530788] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1602.533209] Lustre: Skipped 2 previous similar messages [ 1602.533484] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1602.535037] Lustre: Skipped 1 previous similar message [ 1604.311135] Lustre: server umount lustre-OST0000 complete [ 1606.111903] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1606.114284] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1606.118192] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1607.702968] Lustre: server umount lustre-MDT0000 complete [ 1608.919315] LustreError: 100405:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773234303 with bad export cookie 1573782762209291867 [ 1608.922570] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1608.923724] LustreError: 100405:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1609.051635] Lustre: server umount lustre-MDT0001 complete [ 1612.446707] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 1613.436025] Key type lgssc unregistered [ 1613.555282] LNet: 103624:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1613.557330] LNetError: 103624:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1613.564279] LNet: Removed LNI 192.168.201.137@tcp [ 1613.863630] Key type .llcrypt unregistered [ 1613.864576] Key type ._llcrypt unregistered [ 1623.141701] Key type ._llcrypt registered [ 1623.143295] Key type .llcrypt registered [ 1623.187579] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 1623.534445] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1623.598723] alg: No test for adler32 (adler32-zlib) [ 1624.454624] Lustre: Lustre: Build Version: 2.17.50_236_gaab9274 [ 1624.542783] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 1626.135152] Key type lgssc registered [ 1626.502865] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1629.149603] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1629.158503] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1630.318860] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1631.591248] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1634.388427] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1634.519271] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1635.814119] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1637.520596] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1638.683109] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1640.843625] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1640.928236] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 1640.961825] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1642.825442] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1643.828143] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:67 to 0x240000401:97) [ 1644.590984] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1647.467816] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 1648.197436] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 1649.119823] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1649.121641] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1649.124848] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1650.144616] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1650.145956] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1650.148062] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1655.180083] Lustre: server umount lustre-OST0000 complete [ 1660.384924] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1660.384940] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1660.385730] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1660.385733] Lustre: Skipped 1 previous similar message [ 1660.388673] Lustre: Skipped 1 previous similar message [ 1662.485600] Lustre: server umount lustre-MDT0000 complete [ 1663.725596] LustreError: 107195:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773234358 with bad export cookie 9164426796760528985 [ 1663.729137] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1663.730338] LustreError: 107195:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1663.862614] Lustre: server umount lustre-MDT0001 complete [ 1667.233041] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 1668.236621] Key type lgssc unregistered [ 1668.361344] LNet: 108280:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1668.363979] LNetError: 108280:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1668.372295] LNet: Removed LNI 192.168.201.137@tcp [ 1668.645049] Key type .llcrypt unregistered [ 1668.646115] Key type ._llcrypt unregistered [ 1675.580749] Key type ._llcrypt registered [ 1675.581790] Key type .llcrypt registered [ 1675.617274] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 1676.030938] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1676.065065] alg: No test for adler32 (adler32-zlib) [ 1676.928736] Lustre: Lustre: Build Version: 2.17.50_236_gaab9274 [ 1677.019522] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 1678.607248] Key type lgssc registered [ 1679.006943] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1681.596871] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1681.602459] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1682.756142] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1683.949134] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1686.606255] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1686.735876] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1687.974859] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1689.739554] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1690.949649] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1693.110304] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1693.155366] LustreError: 111052:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1693.161944] LustreError: 111052:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 1693.225459] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1695.000413] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1695.856826] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:99 to 0x240000401:129) [ 1696.727752] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1701.343840] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1701.345791] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1701.350707] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1702.370469] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1702.371091] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1702.376532] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1706.389507] Lustre: server umount lustre-OST0000 complete [ 1712.609254] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1712.609834] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1712.617882] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1712.620206] Lustre: Skipped 2 previous similar messages [ 1713.816618] Lustre: server umount lustre-MDT0000 complete [ 1715.083047] LustreError: 109444:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773234409 with bad export cookie 18286603117601949889 [ 1715.085894] LustreError: 109444:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1715.086064] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1715.193428] Lustre: server umount lustre-MDT0001 complete [ 1718.621023] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 1719.682707] Key type lgssc unregistered [ 1719.813388] LNet: 112639:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1719.816516] LNetError: 112639:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1719.828300] LNet: Removed LNI 192.168.201.137@tcp [ 1720.160121] Key type .llcrypt unregistered [ 1720.161194] Key type ._llcrypt unregistered [ 1728.774335] Key type ._llcrypt registered [ 1728.775802] Key type .llcrypt registered [ 1728.813098] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 1729.127539] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1729.165309] alg: No test for adler32 (adler32-zlib) [ 1730.020501] Lustre: Lustre: Build Version: 2.17.50_236_gaab9274 [ 1730.101323] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 1731.679140] Key type lgssc registered [ 1732.037345] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1734.597388] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1734.604682] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1735.767924] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1737.029065] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1739.720137] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1739.844257] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1741.039171] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1742.805315] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1743.973491] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1746.147035] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1746.251386] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1748.000643] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1749.648665] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1760.613757] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:131 to 0x240000401:161) [ 1767.652668] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 1768.794037] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 1768.796052] Lustre: Skipped 2 previous similar messages [ 1770.093069] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 1770.096518] Lustre: Skipped 2 previous similar messages [ 1774.583968] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 1774.595607] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 1779.534337] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.201.37@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 1779.537498] Lustre: Skipped 2 previous similar messages [ 1782.394583] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.201.37@tcp 192.168.0.[2,10]@tcp [ 1784.823617] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.201.37/24@tcp in log lustre-MDT0000 [ 1784.826414] Lustre: Skipped 2 previous similar messages [ 1791.992334] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.37/24@tcp [ 1791.994030] Lustre: Skipped 1 previous similar message [ 1795.551950] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1795.554358] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1795.558249] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1796.578982] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1796.579031] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1796.579979] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1800.202963] Lustre: server umount lustre-OST0000 complete [ 1801.696665] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1801.700705] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1801.704020] Lustre: Skipped 1 previous similar message [ 1806.303898] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1806.306024] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1806.309529] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1807.577308] Lustre: server umount lustre-MDT0000 complete [ 1808.766079] LustreError: 114730:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773234503 with bad export cookie 7500673452621826119 [ 1808.770560] LustreError: 114730:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1808.771225] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1808.899713] Lustre: server umount lustre-MDT0001 complete [ 1812.270141] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 1813.284348] Key type lgssc unregistered [ 1813.404335] LNet: 118854:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1813.407293] LNetError: 118854:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1813.416310] LNet: Removed LNI 192.168.201.137@tcp [ 1813.719214] Key type .llcrypt unregistered [ 1813.720285] Key type ._llcrypt unregistered [ 1823.567519] Key type ._llcrypt registered [ 1823.568611] Key type .llcrypt registered [ 1823.607702] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 1823.975158] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1824.024522] alg: No test for adler32 (adler32-zlib) [ 1824.885553] Lustre: Lustre: Build Version: 2.17.50_236_gaab9274 [ 1824.973836] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 1826.551176] Key type lgssc registered [ 1826.893512] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1829.391223] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1829.397409] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1830.536041] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 1830.539697] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1830.544176] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.201.37/24@tcp [ 1831.686979] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1834.743450] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1834.878275] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 1834.891928] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.37/24@tcp [ 1834.928840] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1836.889838] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1840.959660] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1844.161346] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1849.957774] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1850.263627] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1854.958662] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1855.423651] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:169 to 0x240000401:193) [ 1860.273882] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1868.836082] hrtimer: interrupt took 2993443 ns [ 1870.822021] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1870.823291] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1870.846091] LustreError: Skipped 1 previous similar message [ 1870.847775] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1870.880119] Lustre: Skipped 1 previous similar message [ 1875.516676] Lustre: server umount lustre-OST0000 complete [ 1881.057352] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1881.057756] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1881.058636] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1881.058641] Lustre: Skipped 1 previous similar message [ 1881.062516] Lustre: Skipped 1 previous similar message [ 1883.627017] Lustre: server umount lustre-MDT0000 complete [ 1885.276208] LustreError: 120354:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773234580 with bad export cookie 6118697071988059821 [ 1885.278959] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1885.280673] LustreError: 120354:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1885.400352] Lustre: server umount lustre-MDT0001 complete [ 1889.433610] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 1890.593457] Key type lgssc unregistered [ 1890.751393] LNet: 123606:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1890.755596] LNetError: 123606:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1890.765024] LNet: Removed LNI 192.168.201.137@tcp [ 1891.122362] Key type .llcrypt unregistered [ 1891.124069] Key type ._llcrypt unregistered [ 1900.425353] Key type ._llcrypt registered [ 1900.426665] Key type .llcrypt registered [ 1900.469811] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_hostid [ 1906.655264] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 1907.092712] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1907.122307] alg: No test for adler32 (adler32-zlib) [ 1907.984539] Lustre: Lustre: Build Version: 2.17.50_236_gaab9274 [ 1908.080797] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 1909.680232] Key type lgssc registered [ 1910.111313] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1924.190764] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1924.201377] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1925.302283] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1925.314651] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1925.354055] Lustre: lustre-MDT0000: new disk, initializing [ 1925.383888] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1925.396723] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1927.045217] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1931.705818] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1931.745763] Lustre: 126945:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1931.770737] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1931.772918] Lustre: Skipped 1 previous similar message [ 1931.809314] Lustre: lustre-MDT0001: new disk, initializing [ 1931.846248] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1931.859465] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1931.864056] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1933.219190] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1936.397958] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1937.591828] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1940.005151] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1940.119808] Lustre: lustre-OST0000: new disk, initializing [ 1940.122717] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1940.152041] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1942.260135] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1945.586552] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1947.018627] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1949.187193] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1949.191685] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1949.222989] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1950.159378] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 1951.517542] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1951.594849] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1954.273549] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1954.274668] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1954.275836] LustreError: Skipped 1 previous similar message [ 1954.279496] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1954.281396] Lustre: Skipped 1 previous similar message [ 1958.357995] Lustre: server umount lustre-OST0000 complete [ 1962.463979] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1962.467508] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1962.472826] Lustre: Skipped 1 previous similar message [ 1962.476910] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1964.512164] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1964.512595] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1965.791046] Lustre: server umount lustre-MDT0000 complete [ 1967.418255] LustreError: 127722:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773234662 with bad export cookie 10527803072537456467 [ 1967.420951] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1967.424236] LustreError: 127722:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1967.546522] Lustre: server umount lustre-MDT0001 complete [ 1970.585557] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1970.771267] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1972.160480] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1975.029142] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1975.154093] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1976.569402] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1978.741305] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1980.270881] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1982.758886] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1985.117303] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1987.069960] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1988.551499] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1995.773748] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 7 sec [ 1997.192751] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1997.280606] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2000.790865] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2000.849991] Lustre: lustre-OST0001: new disk, initializing [ 2000.851571] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2000.878309] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2000.881155] Lustre: Skipped 1 previous similar message [ 2002.883293] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 2006.345466] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2009.600435] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2009.605798] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2009.639548] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2010.497256] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 4 sec [ 2011.891152] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2011.972448] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2013.276984] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8967c7825000.ost_server_uuid 50 [ 2013.811543] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8967c7825000.ost_server_uuid in FULL state after 0 sec [ 2019.809195] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2019.809566] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2019.814027] LustreError: Skipped 1 previous similar message [ 2019.819509] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2019.823174] Lustre: Skipped 2 previous similar messages [ 2021.825996] Lustre: server umount lustre-OST0001 complete [ 2024.930136] LustreError: 132369:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2024.930742] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2024.932192] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2024.936206] LustreError: 132369:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 3 previous similar messages [ 2024.951223] Lustre: Skipped 2 previous similar messages [ 2029.528654] Lustre: server umount lustre-OST0000 complete [ 2031.584219] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2035.168246] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2035.169250] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2035.171520] Lustre: Skipped 2 previous similar messages [ 2035.179468] Lustre: Skipped 1 previous similar message [ 2045.407679] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2045.483170] Lustre: server umount lustre-MDT0000 complete [ 2047.033284] LustreError: 135005:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773234741 with bad export cookie 10527803072537457335 [ 2047.038059] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2047.038456] LustreError: 135005:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2047.169962] Lustre: server umount lustre-MDT0001 complete [ 2055.814299] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 2057.079606] Key type lgssc unregistered [ 2057.237358] LNet: 136687:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2057.239968] LNetError: 136687:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2057.246324] LNet: Removed LNI 192.168.201.137@tcp [ 2057.608782] Key type .llcrypt unregistered [ 2057.610642] Key type ._llcrypt unregistered [ 2074.033218] Key type ._llcrypt registered [ 2074.034638] Key type .llcrypt registered [ 2074.088845] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_hostid [ 2080.877330] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 2081.386966] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2081.415992] alg: No test for adler32 (adler32-zlib) [ 2082.281463] Lustre: Lustre: Build Version: 2.17.50_236_gaab9274 [ 2082.384096] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 2083.975162] Key type lgssc registered [ 2084.447539] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2099.748760] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2099.759309] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2100.878181] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2100.907117] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2100.962619] Lustre: lustre-MDT0000: new disk, initializing [ 2101.018296] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2101.029328] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2102.925873] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 2108.442787] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2108.502649] Lustre: 140777:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2108.538417] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2108.542229] Lustre: Skipped 1 previous similar message [ 2108.586925] Lustre: lustre-MDT0001: new disk, initializing [ 2108.627777] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2108.646311] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2108.652403] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2110.346394] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 2114.012490] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2115.664725] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2118.794458] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2118.940130] Lustre: lustre-OST0000: new disk, initializing [ 2118.944758] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2118.988787] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2121.535420] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 2124.296591] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2124.300823] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2124.347524] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2125.234559] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2127.423041] Lustre: Failing over lustre-OST0000 [ 2127.512210] Lustre: server umount lustre-OST0000 complete [ 2141.245605] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2141.417926] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2141.435442] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2141.444443] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2142.135378] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2144.050885] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 2145.567909] Lustre: 139006:0:(client.c:2516:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773234824/real 1773234824] req@000000007da672c9 x1859371429555200/t0(0) o13->lustre-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 1 dl 1773234840 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2145.584352] Lustre: 139006:0:(client.c:2516:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2145.804750] Lustre: Failing over lustre-MDT0000 [ 2145.924607] Lustre: server umount lustre-MDT0000 complete [ 2146.787287] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2146.793637] Lustre: Skipped 2 previous similar messages [ 2150.879123] Lustre: 139007:0:(client.c:2516:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773234829/real 1773234829] req@000000007e704c78 x1859371429556480/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1773234845 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2150.891376] Lustre: 139007:0:(client.c:2516:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 2151.905104] LustreError: 140790:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2151.917697] LustreError: 140790:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 7 previous similar messages [ 2155.935213] Lustre: 139008:0:(client.c:2516:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773234834/real 1773234834] req@000000000b6ea565 x1859371429557504/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1773234850 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2157.027038] LustreError: 141561:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2157.034644] LustreError: 141561:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 3 previous similar messages [ 2162.152580] LustreError: 140785:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2162.173276] LustreError: 140785:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 3 previous similar messages [ 2163.174228] Lustre: 139006:0:(client.c:2516:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773234841/real 1773234841] req@000000007da672c9 x1859371429562880/t0(0) o400->MGC192.168.201.137@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1773234857 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2163.213499] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2163.227826] Lustre: 140782:0:(mgc_request.c:1917:mgc_process_log()) MGC192.168.201.137@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 2164.909433] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2167.784198] LustreError: 140787:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2167.819156] LustreError: 140787:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 4 previous similar messages [ 2172.946391] LustreError: 140787:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2172.971225] LustreError: 140787:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 3 previous similar messages [ 2173.413442] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xaa1c3447944333c3 [ 2173.425193] LustreError: 144621:0:(ldlm_resource.c:1170:ldlm_resource_complain()) MGC192.168.201.137@tcp: namespace resource [0x65727473756c:0x5:0x0].0x0 (0000000010826178) refcount nonzero (2) after lock cleanup; forcing cleanup. [ 2173.439551] Lustre: MGC192.168.201.137@tcp: Connection restored to 0@lo (at 0@lo) [ 2173.641223] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -114 [ 2173.705546] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2173.723466] Lustre: lustre-OST0000: Recovery over after 0:31, of 3 clients 3 recovered and 0 were evicted. [ 2173.778529] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2174.769709] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2178.868704] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 2179.076771] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2179.087971] Lustre: Skipped 2 previous similar messages [ 2179.108838] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 2179.145519] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 2189.283968] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2189.294809] Lustre: Skipped 1 previous similar message [ 2189.298148] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2189.299099] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2189.311532] Lustre: Skipped 1 previous similar message [ 2193.594994] Lustre: server umount lustre-OST0000 complete [ 2199.521464] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2199.526356] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2199.535375] Lustre: Skipped 1 previous similar message [ 2199.538591] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2204.621809] Lustre: server umount lustre-MDT0000 complete [ 2204.645309] LustreError: 141561:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2204.687287] LustreError: 141561:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 4 previous similar messages [ 2209.720193] LustreError: 141555:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773234904 with bad export cookie 12257729767830401987 [ 2209.728492] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2209.737194] LustreError: 141555:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2210.536964] Lustre: server umount lustre-MDT0001 complete [ 2225.634545] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 2228.360231] Key type lgssc unregistered [ 2228.708680] LNet: 146019:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2228.717621] LNetError: 146019:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2229.734519] LNet: Removed LNI 192.168.201.137@tcp [ 2230.478311] Key type .llcrypt unregistered [ 2230.480051] Key type ._llcrypt unregistered [ 2262.461180] Key type ._llcrypt registered [ 2262.463298] Key type .llcrypt registered [ 2262.545525] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 2263.578327] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2263.629225] alg: No test for adler32 (adler32-zlib) [ 2264.669373] Lustre: Lustre: Build Version: 2.17.50_236_gaab9274 [ 2264.842436] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 2266.487804] Key type lgssc registered [ 2267.294663] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2274.172391] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2274.187934] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2275.631739] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2279.109615] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 2285.460723] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2286.056207] LustreError: 147521:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2286.067586] LustreError: 147521:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 2288.827342] LustreError: 147550:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2288.834922] LustreError: 147550:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2288.839291] LustreError: 147550:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 13, retries 0, failed: rc = -5 [ 2289.084508] Lustre: server umount lustre-MDT0000 complete [ 2300.589391] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_hostid [ 2309.218179] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 2340.983579] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2341.244769] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2341.277862] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2341.370218] Lustre: lustre-MDT0000: new disk, initializing [ 2341.464894] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2341.484565] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2345.090995] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 2353.896484] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2354.026434] Lustre: 150708:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2354.061563] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2354.063712] Lustre: Skipped 1 previous similar message [ 2354.126826] Lustre: lustre-MDT0001: new disk, initializing [ 2354.196793] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2354.219284] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2354.226952] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2357.329349] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 2363.725582] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2366.428844] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2371.187518] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2371.432315] Lustre: lustre-OST0000: new disk, initializing [ 2371.436169] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2371.546811] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2375.859804] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 2380.319718] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2380.325428] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2380.396917] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2381.321830] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2385.331428] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 2386.518561] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 09:17:59 (1773235079) [ 2594.789681] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2594.797584] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2594.804526] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2596.901721] Lustre: server umount lustre-MDT0000 complete [ 2599.882091] LustreError: 150698:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773235294 with bad export cookie 11676418503828814096 [ 2599.885000] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2599.901243] LustreError: 150698:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2600.415899] LustreError: 153455:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2600.417263] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2600.445950] Lustre: Skipped 2 previous similar messages [ 2600.451106] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2600.453784] Lustre: Skipped 3 previous similar messages [ 2604.447679] LustreError: 150713:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2604.448037] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2609.569090] LustreError: 154108:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2609.569720] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2614.239118] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2614.410084] Lustre: server umount lustre-MDT0001 complete [ 2623.425204] Lustre: server umount lustre-OST0000 complete [ 2630.929670] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_hostid [ 2638.200506] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 2664.243028] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2664.439659] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2664.461790] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2664.531079] Lustre: lustre-MDT0000: new disk, initializing [ 2664.602104] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2664.616343] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2667.545352] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 2675.057813] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2675.192901] Lustre: 156953:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2675.236474] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2675.243326] Lustre: Skipped 1 previous similar message [ 2675.337279] Lustre: lustre-MDT0001: new disk, initializing [ 2675.416896] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2675.445202] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2675.458587] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2678.058821] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 2683.283496] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2685.323093] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2689.295458] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2689.476539] Lustre: lustre-OST0000: new disk, initializing [ 2689.480971] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2694.020896] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 2698.257804] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2698.272135] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2698.348412] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2699.792281] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2702.687404] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2702.844233] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2705.586673] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2705.713787] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2708.448652] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2708.456281] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2708.460759] LustreError: Skipped 1 previous similar message [ 2708.476330] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2708.482334] Lustre: Skipped 1 previous similar message [ 2713.396963] Lustre: server umount lustre-OST0000 complete [ 2716.644230] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2716.655976] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2716.666771] Lustre: Skipped 1 previous similar message [ 2716.671530] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2722.685083] Lustre: server umount lustre-MDT0000 complete [ 2723.825109] LustreError: 157737:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2723.848809] LustreError: 157737:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 2725.531596] LustreError: 156947:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773235420 with bad export cookie 11676418503828972919 [ 2725.536664] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2725.543963] LustreError: 156947:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2725.804581] Lustre: server umount lustre-MDT0001 complete [ 2732.707885] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 09:23:45 (1773235425) [ 2745.463750] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_hostid [ 2753.665019] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 2781.723319] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2781.982393] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2782.010105] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2782.074137] Lustre: lustre-MDT0000: new disk, initializing [ 2782.150876] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2782.157780] Lustre: Skipped 1 previous similar message [ 2782.172861] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2785.597442] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 2794.771174] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2794.900379] Lustre: 163266:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2795.126905] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2798.228119] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 2803.373763] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2805.920388] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2810.617890] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2814.665368] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 2818.075790] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2818.082573] Lustre: Skipped 1 previous similar message [ 2818.086096] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2818.148424] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2819.914390] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2828.263042] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2828.267348] Lustre: Skipped 1 previous similar message [ 2828.270106] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2828.272078] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2828.272086] Lustre: Skipped 2 previous similar messages [ 2828.290519] LustreError: Skipped 1 previous similar message [ 2830.392675] Lustre: server umount lustre-OST0000 complete [ 2839.443404] Lustre: server umount lustre-MDT0000 complete [ 2841.297633] LustreError: 165014:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773235536 with bad export cookie 11676418503828973808 [ 2841.300491] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2841.311369] LustreError: 165014:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2841.507981] Lustre: server umount lustre-MDT0001 complete [ 2846.361912] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 09:25:39 (1773235539) [ 2856.226087] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_hostid [ 2862.642234] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 2885.214950] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2885.394153] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2885.434410] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2885.439821] Lustre: Skipped 3 previous similar messages [ 2885.490649] Lustre: lustre-MDT0000: new disk, initializing [ 2885.493908] Lustre: Skipped 2 previous similar messages [ 2885.564490] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2885.568250] Lustre: Skipped 2 previous similar messages [ 2885.581048] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2887.979090] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 2895.378589] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2895.489357] Lustre: 169182:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2895.684125] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2898.377205] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 2903.899167] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2906.474301] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2910.518905] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2914.749692] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 2919.904724] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2920.559907] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2930.662812] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2930.663575] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2930.668375] LustreError: Skipped 1 previous similar message [ 2930.673748] Lustre: Skipped 3 previous similar messages [ 2930.674983] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2930.684431] Lustre: Skipped 6 previous similar messages [ 2935.302135] Lustre: server umount lustre-OST0000 complete [ 2944.218213] Lustre: server umount lustre-MDT0000 complete [ 2946.020587] LustreError: 169195:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2946.035859] LustreError: 169195:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 2946.766837] LustreError: 169174:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773235641 with bad export cookie 11676418503828974886 [ 2946.772155] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2946.773645] LustreError: 169174:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2947.002920] Lustre: server umount lustre-MDT0001 complete [ 2952.367459] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 2953.847130] Key type lgssc unregistered [ 2954.004632] LNet: 172660:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2954.008885] LNetError: 172660:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2954.016640] LNet: Removed LNI 192.168.201.137@tcp [ 2954.415729] Key type .llcrypt unregistered [ 2954.417482] Key type ._llcrypt unregistered [ 2967.271344] Key type ._llcrypt registered [ 2967.273497] Key type .llcrypt registered [ 2967.344802] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 2967.996053] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2968.005809] alg: No test for adler32 (adler32-zlib) [ 2968.943707] Lustre: Lustre: Build Version: 2.17.50_236_gaab9274 [ 2969.070367] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 2970.679360] Key type lgssc registered [ 2971.486564] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2977.585529] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2977.601796] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2979.040639] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2982.095852] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 2987.890677] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2988.153896] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2990.681751] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 2994.731235] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2997.600797] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3002.500822] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3002.778217] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3003.822459] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 3006.344251] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 3010.130924] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3033.577031] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3033.577593] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3033.596088] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3033.609566] Lustre: Skipped 1 previous similar message [ 3035.871607] Lustre: server umount lustre-OST0000 complete [ 3038.689033] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3038.704885] Lustre: Skipped 1 previous similar message [ 3038.709543] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3038.714393] Lustre: Skipped 1 previous similar message [ 3043.809373] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3043.816326] Lustre: Skipped 1 previous similar message [ 3044.500662] Lustre: server umount lustre-MDT0000 complete [ 3047.086498] LustreError: 174031:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773235741 with bad export cookie 9739085507372864017 [ 3047.093921] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3047.097709] LustreError: 174031:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3047.376393] Lustre: server umount lustre-MDT0001 complete [ 3054.487608] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 3056.151693] Key type lgssc unregistered [ 3056.335688] LNet: 177251:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3056.341591] LNetError: 177251:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3056.349735] LNet: Removed LNI 192.168.201.137@tcp [ 3056.878978] Key type .llcrypt unregistered [ 3056.880891] Key type ._llcrypt unregistered [ 3072.569022] Key type ._llcrypt registered [ 3072.571213] Key type .llcrypt registered [ 3072.642164] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 3073.438640] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3073.449026] alg: No test for adler32 (adler32-zlib) [ 3074.466981] Lustre: Lustre: Build Version: 2.17.50_236_gaab9274 [ 3074.649404] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 3076.295197] Key type lgssc registered [ 3077.233433] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3082.886599] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3082.904613] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3084.317457] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3087.056028] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 3092.625451] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3092.890896] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3096.072964] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 3100.558399] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3103.328633] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3108.786323] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3109.105827] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3110.128673] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 3113.490066] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 3117.849754] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3122.661801] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3122.669226] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3122.671893] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3122.682079] Lustre: Skipped 1 previous similar message [ 3124.672672] Lustre: lustre-OST0000: Not available for connect from 192.168.201.37@tcp (stopping) [ 3127.779415] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3127.788092] Lustre: Skipped 1 previous similar message [ 3128.584497] Lustre: server umount lustre-OST0000 complete [ 3132.189245] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff8967c4cab800.ost_server_uuid 50 [ 3133.252615] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 3144.705888] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3144.716993] Lustre: Skipped 1 previous similar message [ 3144.720471] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3149.284180] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3149.288404] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3149.295931] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3150.886645] Lustre: server umount lustre-MDT0000 complete [ 3152.810567] LustreError: 178613:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773235847 with bad export cookie 4700999124500223591 [ 3152.816098] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3152.819554] LustreError: 178613:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3152.964897] Lustre: server umount lustre-MDT0001 complete [ 3157.473765] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 09:30:51 (1773235851) [ 3160.744450] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3161.017215] LustreError: 182111:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3161.026358] LustreError: 182111:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 3161.074737] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3163.510365] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 3166.176622] LustreError: 182112:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3169.018253] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3169.258982] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3172.248258] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 3175.852568] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3178.493401] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3183.280891] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3183.511947] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3185.578778] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 3187.279857] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 3190.739690] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3195.164656] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3195.298980] Lustre: lustre-OST0001: new disk, initializing [ 3195.304770] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3199.037133] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 3199.191912] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3199.198641] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3199.261851] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3203.584468] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3206.118386] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3206.121984] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3206.132568] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3206.138514] Lustre: Skipped 1 previous similar message [ 3211.289120] Lustre: server umount lustre-OST0000 complete [ 3214.253619] LustreError: 183884:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3214.629518] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3214.748414] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 3217.078417] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3217.205899] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 3219.367598] LustreError: 183706:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3219.377599] LustreError: 183706:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 3224.486775] LustreError: 183884:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3224.494451] LustreError: 183884:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 3230.177281] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3230.178034] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3230.181804] LustreError: Skipped 1 previous similar message [ 3230.187235] Lustre: Skipped 1 previous similar message [ 3230.190111] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3230.196558] Lustre: Skipped 4 previous similar messages [ 3233.962591] Lustre: server umount lustre-OST0001 complete [ 3240.418109] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3240.429165] Lustre: Skipped 1 previous similar message [ 3242.308278] Lustre: server umount lustre-MDT0000 complete [ 3244.379551] LustreError: 182092:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773235939 with bad export cookie 4700999124500224662 [ 3244.384369] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3244.390065] LustreError: 182092:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3244.565603] Lustre: server umount lustre-MDT0001 complete [ 3258.790583] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 09:32:32 (1773235952) [ 3262.189710] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3262.346267] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3262.368521] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3262.502401] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3262.506660] Lustre: Skipped 1 previous similar message [ 3264.504577] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 3269.263161] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3269.343269] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3269.371239] Lustre: 187882:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3272.219928] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 3275.612662] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3277.683228] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3281.839207] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3282.005321] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3286.033742] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 3289.597237] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3290.707095] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 3293.954019] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3294.034472] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3297.898312] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 3301.200661] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3305.953583] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3305.953993] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3305.957598] LustreError: Skipped 1 previous similar message [ 3305.960124] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3305.960131] Lustre: Skipped 2 previous similar messages [ 3305.975312] Lustre: Skipped 1 previous similar message [ 3308.948089] Lustre: server umount lustre-OST0000 complete [ 3311.590910] LustreError: 190128:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3311.603727] LustreError: 190128:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 5 previous similar messages [ 3321.829476] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3321.835083] LustreError: Skipped 1 previous similar message [ 3326.972847] Lustre: server umount lustre-OST0001 complete [ 3335.283526] Lustre: server umount lustre-MDT0000 complete [ 3337.179623] LustreError: 187874:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773236031 with bad export cookie 4700999124500226062 [ 3337.186291] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3337.189537] LustreError: 187874:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3337.377175] Lustre: server umount lustre-MDT0001 complete [ 3350.568830] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 09:34:04 (1773236044) [ 3359.805340] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_hostid [ 3365.659669] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 3387.635743] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3387.804713] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3387.830844] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3387.890304] Lustre: lustre-MDT0000: new disk, initializing [ 3387.959720] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3387.965649] Lustre: Skipped 3 previous similar messages [ 3387.980525] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3390.243346] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 3396.582658] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3396.658381] Lustre: 195459:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3396.700689] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3396.703662] Lustre: Skipped 1 previous similar message [ 3396.749722] Lustre: lustre-MDT0001: new disk, initializing [ 3396.869039] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3396.883900] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3399.412757] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 3404.409627] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3406.722508] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3411.459179] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3411.639814] Lustre: lustre-OST0000: new disk, initializing [ 3411.643218] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3415.127620] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 3418.648076] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3418.658056] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3418.726681] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3419.923952] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3422.671168] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3422.797666] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3425.505632] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3425.682347] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3428.837715] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3428.837982] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3428.840906] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3428.840914] Lustre: Skipped 9 previous similar messages [ 3428.846367] Lustre: Skipped 5 previous similar messages [ 3428.861394] LustreError: Skipped 2 previous similar messages [ 3441.119234] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3441.229629] Lustre: server umount lustre-OST0000 complete [ 3449.422341] Lustre: server umount lustre-MDT0000 complete [ 3451.290145] LustreError: 195450:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773236146 with bad export cookie 4700999124500227609 [ 3451.292569] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3451.295341] LustreError: 195450:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3451.524658] Lustre: server umount lustre-MDT0001 complete [ 3456.167215] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3456.471500] LustreError: 199270:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3456.492048] LustreError: 199270:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 5 previous similar messages [ 3456.533564] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3456.536296] Lustre: Skipped 2 previous similar messages [ 3458.773614] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 3464.364956] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3467.238687] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 3470.779772] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3472.944840] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3477.130710] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3480.654731] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 3484.346869] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3486.986541] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3487.165947] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3489.545309] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3490.710224] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 1 sec [ 3491.808099] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3494.884097] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3494.894722] Lustre: Skipped 3 previous similar messages [ 3497.838282] Lustre: server umount lustre-OST0000 complete [ 3502.001728] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3505.035204] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3530.362217] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3533.593725] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 3534.693037] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3534.699680] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3536.514476] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3544.934407] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3544.941277] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 3551.097843] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 13 sec [ 3558.920540] Lustre: server umount lustre-OST0000 complete [ 3561.952049] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3561.955410] LustreError: Skipped 3 previous similar messages [ 3561.958710] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3561.962271] Lustre: Skipped 14 previous similar messages [ 3575.263176] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3575.386817] Lustre: server umount lustre-MDT0000 complete [ 3575.777216] LustreError: 203414:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3575.791348] LustreError: 203414:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 3577.743429] LustreError: 199252:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773236272 with bad export cookie 4700999124500228491 [ 3577.749157] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3577.754861] LustreError: 199252:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3577.945776] Lustre: server umount lustre-MDT0001 complete [ 3583.267292] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 09:37:56 (1773236276) [ 3587.521232] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3587.913925] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3587.924254] Lustre: Skipped 3 previous similar messages [ 3590.354830] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 3596.046992] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3598.662554] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 3601.775090] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3603.943725] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3608.408117] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3612.331354] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 3615.506366] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3617.669949] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3617.779691] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3619.966991] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3622.171625] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 2 sec [ 3626.339928] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3626.448153] Lustre: lustre-OST0001: new disk, initializing [ 3626.451862] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3629.890802] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 3633.713404] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3633.725105] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3633.784719] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3634.786565] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3637.054624] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3637.182409] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3639.493924] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3639.632143] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3643.877177] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3643.886820] Lustre: Skipped 5 previous similar messages [ 3655.135109] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3655.204815] Lustre: server umount lustre-OST0001 complete [ 3658.749282] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3658.891418] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 3659.239406] LustreError: 206340:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3659.255094] LustreError: 206340:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 3 previous similar messages [ 3661.260778] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3661.379701] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 3686.656798] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3690.180859] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 3692.008772] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3692.017930] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3692.018403] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3692.028212] Lustre: Skipped 1 previous similar message [ 3693.220672] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3694.174977] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3697.120880] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3697.128165] LustreError: Skipped 3 previous similar messages [ 3701.474358] Lustre: server umount lustre-OST0001 complete [ 3721.534578] Lustre: server umount lustre-OST0000 complete [ 3738.079697] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3738.185314] Lustre: server umount lustre-MDT0000 complete [ 3740.370257] LustreError: 206361:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773236435 with bad export cookie 4700999124500229443 [ 3740.373252] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3740.380301] LustreError: 206361:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3746.596247] Lustre: server umount lustre-MDT0001 complete [ 3759.059773] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 09:40:52 (1773236452) [ 3762.407770] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3762.546065] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3762.569239] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3764.638460] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 3769.099987] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3769.173424] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3769.195757] Lustre: 212715:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3771.437218] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 3774.116677] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3775.902305] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3779.292305] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3779.412264] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3782.342210] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 3785.234132] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3795.548548] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3795.609265] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3798.497417] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 3801.184436] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3802.075503] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3804.151092] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3804.264380] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3806.284121] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3806.389134] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3808.306631] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8967c4096000.ost_server_uuid 50 [ 3809.173334] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8967c4096000.ost_server_uuid in FULL state after 0 sec [ 3809.724041] Lustre: Permanently deactivating lustre-OST0001 [ 3809.733784] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 3811.117377] Lustre: Permanently reactivating lustre-OST0001 [ 3818.470458] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3818.474132] Lustre: Skipped 28 previous similar messages [ 3818.970446] Lustre: server umount lustre-OST0001 complete [ 3823.590201] LustreError: 214901:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3823.612185] LustreError: 214901:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 25 previous similar messages [ 3835.360076] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3835.459400] Lustre: server umount lustre-OST0000 complete [ 3843.507691] Lustre: server umount lustre-MDT0000 complete [ 3845.316205] LustreError: 213441:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773236540 with bad export cookie 4700999124500230717 [ 3845.316904] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3845.326124] LustreError: 213441:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3845.488074] Lustre: server umount lustre-MDT0001 complete [ 3856.894854] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 09:42:30 (1773236550) [ 3860.727231] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3860.897537] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3860.922613] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3860.927748] Lustre: Skipped 5 previous similar messages [ 3861.051736] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3861.056033] Lustre: Skipped 8 previous similar messages [ 3863.092266] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 3867.234224] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3867.299142] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3867.320493] Lustre: 218817:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3869.305277] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 3872.051239] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3874.038926] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3877.404424] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3877.554359] Lustre: Permanently deactivating lustre-OST0000 [ 3877.561883] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 3880.303679] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 3882.809420] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3883.010254] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 3886.280271] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3889.780820] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 3891.268838] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 3892.845400] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3893.943963] Lustre: Permanently reactivating lustre-OST0000 [ 3893.954309] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 3893.961157] Lustre: Skipped 2 previous similar messages [ 3898.894964] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 3898.900307] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3898.907352] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3898.926720] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 3898.938690] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 3905.002663] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3905.009322] Lustre: Skipped 13 previous similar messages [ 3908.877788] Lustre: server umount lustre-OST0001 complete [ 3917.105259] Lustre: server umount lustre-OST0000 complete [ 3925.447376] Lustre: server umount lustre-MDT0000 complete [ 3927.506636] LustreError: 218812:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773236622 with bad export cookie 4700999124500232369 [ 3927.509650] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3927.512790] LustreError: 218812:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3927.687954] Lustre: server umount lustre-MDT0001 complete [ 3933.787230] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 3935.373213] Key type lgssc unregistered [ 3935.569813] LNet: 223203:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3935.573960] LNetError: 223203:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3935.583948] LNet: Removed LNI 192.168.201.137@tcp [ 3936.007135] Key type .llcrypt unregistered [ 3936.009118] Key type ._llcrypt unregistered [ 3946.983086] Key type ._llcrypt registered [ 3946.984671] Key type .llcrypt registered [ 3947.036493] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 3947.510123] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3947.569292] alg: No test for adler32 (adler32-zlib) [ 3948.472302] Lustre: Lustre: Build Version: 2.17.50_236_gaab9274 [ 3948.577212] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 3950.175198] Key type lgssc registered [ 3950.777840] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3955.597551] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3955.609330] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3956.868110] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3958.832554] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 3962.710945] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3962.795409] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3962.819976] Lustre: 224619:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3962.828125] Lustre: Permanently deactivating lustre-MDT0001 [ 3962.839721] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 3963.012815] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3964.672610] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 3967.202623] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3968.980113] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3971.566814] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 3972.261658] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3972.472260] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3975.185544] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 3978.068991] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3981.528022] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3981.672378] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 3981.680119] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3984.507870] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 3985.471199] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 3987.563596] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 3987.564070] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:193) [ 3987.584256] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3990.406549] Lustre: Permanently reactivating lustre-MDT0001 [ 3990.414861] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 3990.418909] Lustre: Skipped 2 previous similar messages [ 3991.833491] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3991.841419] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 3991.846918] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 3991.854297] LustreError: 227751:0:(llog_osd.c:2259:llog_osd_get_cat_list()) lustre-MDT0001-osp-MDT0000: error reading CATALOGS: rc = -108 [ 3991.854995] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 3991.859059] LustreError: 227751:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -108 [ 3991.867355] LustreError: 227751:0:(obd_config.c:2136:class_config_llog_handler()) MGC192.168.201.137@tcp: cfg command failed: rc = -108 [ 3991.871473] Lustre: cmd=cf00f 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001-osp-MDT0000.active=1 [ 3991.871473] [ 3991.875659] LustreError: 224623:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 3997.617378] Lustre: lustre-MDT0001: Client a35c02f2-b629-433d-9dc3-27b00c431db8 (at 192.168.201.37@tcp) reconnecting [ 3999.226830] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3999.953945] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4002.168367] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 4002.278865] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 4004.319842] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 4004.431639] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 4005.758319] Lustre: Permanently deactivating lustre-MDT0001 [ 4005.775511] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 4005.785079] Lustre: Skipped 2 previous similar messages [ 4011.532584] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 4017.122487] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 4017.124893] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4017.131250] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4018.658885] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4018.659685] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4018.664067] Lustre: Skipped 1 previous similar message [ 4018.669968] Lustre: Skipped 1 previous similar message [ 4021.424367] Lustre: server umount lustre-MDT0000 complete [ 4023.276517] LustreError: 226266:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773236718 with bad export cookie 8058097367273935030 [ 4023.280596] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4023.283774] LustreError: 226266:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4023.443846] Lustre: server umount lustre-MDT0001 complete [ 4031.364911] Lustre: server umount lustre-OST0000 complete [ 4036.370435] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 09:45:30 (1773236730) [ 4041.062894] Lustre: server umount lustre-OST0001 complete [ 4043.821925] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_hostid [ 4048.008613] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 4064.106594] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4064.241961] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4064.248335] Lustre: Skipped 2 previous similar messages [ 4064.262152] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4064.318697] Lustre: lustre-MDT0000: new disk, initializing [ 4064.365261] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4064.377891] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4066.285931] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 4071.687153] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4071.746408] Lustre: 232318:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4071.778520] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4071.781930] Lustre: Skipped 1 previous similar message [ 4071.835722] Lustre: lustre-MDT0001: new disk, initializing [ 4071.897583] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4071.902572] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4073.660596] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 4077.296836] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4078.823901] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4081.828591] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4081.994511] Lustre: lustre-OST0000: new disk, initializing [ 4081.998593] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4082.052927] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4082.057068] Lustre: Skipped 1 previous similar message [ 4084.680394] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 4088.330027] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4088.335297] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4088.399288] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4088.680432] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4089.831445] LustreError: 234567:0:(mdt_handler.c:3251:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 4095.138032] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4095.223300] Lustre: lustre-OST0001: new disk, initializing [ 4095.225646] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4097.935443] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4097.941327] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4097.993350] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4098.136135] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 4099.927122] LustreError: 234567:0:(mdt_handler.c:3251:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 4102.524856] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4103.013890] LustreError: 234567:0:(mdt_handler.c:3251:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 4103.394897] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4113.111165] LustreError: 234567:0:(mdt_handler.c:3251:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 4117.937134] Lustre: lustre-OST0001: Not available for connect from 192.168.201.37@tcp (stopping) [ 4117.941282] Lustre: Skipped 2 previous similar messages [ 4118.501085] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4118.502862] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4118.505461] LustreError: Skipped 1 previous similar message [ 4118.511165] Lustre: Skipped 1 previous similar message [ 4120.134814] Lustre: server umount lustre-OST0001 complete [ 4123.623685] LustreError: 235371:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4123.633746] LustreError: 235371:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 3 previous similar messages [ 4128.741516] LustreError: 235371:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4128.751517] LustreError: 235371:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 4133.858223] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4133.859082] LustreError: 235371:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4133.859785] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4133.859791] Lustre: Skipped 2 previous similar messages [ 4133.862681] Lustre: Skipped 2 previous similar messages [ 4133.869536] LustreError: 235371:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 4138.980143] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4138.980676] LustreError: 234048:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4138.985502] Lustre: Skipped 2 previous similar messages [ 4138.998518] LustreError: 234048:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 4139.172246] Lustre: server umount lustre-OST0000 complete [ 4143.583923] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4143.587473] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4149.217756] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4149.221940] Lustre: Skipped 4 previous similar messages [ 4155.359104] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4155.462392] Lustre: server umount lustre-MDT0000 complete [ 4157.184227] LustreError: 234062:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773236851 with bad export cookie 8058097367273938075 [ 4157.188469] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4157.190257] LustreError: 234062:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4163.437993] Lustre: server umount lustre-MDT0001 complete [ 4168.425261] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 4169.803229] Key type lgssc unregistered [ 4169.970497] LNet: 236965:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4169.974329] LNetError: 236965:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4169.985284] LNet: Removed LNI 192.168.201.137@tcp [ 4170.397538] Key type .llcrypt unregistered [ 4170.398479] Key type ._llcrypt unregistered [ 4189.261237] Key type ._llcrypt registered [ 4189.263237] Key type .llcrypt registered [ 4189.320919] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 4189.864477] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4189.915725] alg: No test for adler32 (adler32-zlib) [ 4190.831547] Lustre: Lustre: Build Version: 2.17.50_236_gaab9274 [ 4190.946325] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 4192.543149] Key type lgssc registered [ 4193.058057] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4196.689076] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4196.698148] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4197.867029] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4197.883110] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4197.968453] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4199.860888] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 4203.706072] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4203.754401] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4203.769119] Lustre: 239129:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4203.902179] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4205.635192] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 4207.997294] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4209.472192] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4212.401624] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4212.497525] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4212.588478] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4215.256753] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 4216.868551] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 4218.061996] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4262.884522] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4262.887809] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4262.889309] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4262.894374] Lustre: Skipped 1 previous similar message [ 4262.899270] Lustre: Skipped 1 previous similar message [ 4265.979867] Lustre: server umount lustre-OST0000 complete [ 4267.837406] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 4272.839890] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4273.015097] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4275.321836] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 4277.678935] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4282.216875] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4282.224335] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4282.224354] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4282.233277] Lustre: Skipped 1 previous similar message [ 4313.060350] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4313.060722] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4313.065152] LustreError: Skipped 1 previous similar message [ 4313.065824] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4313.072038] Lustre: Skipped 1 previous similar message [ 4318.192173] Lustre: server umount lustre-OST0000 complete [ 4321.761211] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4321.766355] LustreError: Skipped 1 previous similar message [ 4321.768496] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4321.775578] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4321.778481] Lustre: Skipped 1 previous similar message [ 4328.416540] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4328.420927] Lustre: Skipped 3 previous similar messages [ 4332.512806] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4332.516150] Lustre: Skipped 1 previous similar message [ 4334.047272] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4334.129157] Lustre: server umount lustre-MDT0000 complete [ 4335.697640] LustreError: 239123:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773237030 with bad export cookie 8078474777208866071 [ 4335.699612] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4335.700710] LustreError: 239123:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4335.852560] Lustre: server umount lustre-MDT0001 complete [ 4340.287592] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 4341.652172] Key type lgssc unregistered [ 4341.814509] LNet: 244084:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4341.820352] LNetError: 244084:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4341.830319] LNet: Removed LNI 192.168.201.137@tcp [ 4342.210616] Key type .llcrypt unregistered [ 4342.212235] Key type ._llcrypt unregistered [ 4352.778200] Key type ._llcrypt registered [ 4352.780731] Key type .llcrypt registered [ 4352.829126] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 4353.280458] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4353.331621] alg: No test for adler32 (adler32-zlib) [ 4354.231646] Lustre: Lustre: Build Version: 2.17.50_236_gaab9274 [ 4354.344866] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 4355.943306] Key type lgssc registered [ 4356.443965] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4359.902571] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4359.916615] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4361.128513] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4363.047362] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 4367.211209] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4367.375301] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4369.177644] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 4371.589707] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4373.134910] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4375.907699] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4376.068417] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4378.468457] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 4380.734686] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4385.320649] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 4395.489342] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4395.489892] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4395.491416] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4395.501604] Lustre: Skipped 1 previous similar message [ 4400.609074] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4400.612953] Lustre: Skipped 2 previous similar messages [ 4401.199817] Lustre: server umount lustre-OST0000 complete [ 4403.167989] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4403.171373] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4403.178225] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4405.729725] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4405.730294] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4409.261595] Lustre: server umount lustre-MDT0000 complete [ 4410.848279] LustreError: 245460:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4410.857506] LustreError: 245460:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 3 previous similar messages [ 4410.902296] LustreError: 245441:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773237105 with bad export cookie 459018459771133066 [ 4410.905792] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4410.907672] LustreError: 245441:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4411.060772] Lustre: server umount lustre-MDT0001 complete [ 4415.467873] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 4416.687363] Key type lgssc unregistered [ 4416.832663] LNet: 249410:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4416.836565] LNetError: 249410:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4416.845309] LNet: Removed LNI 192.168.201.137@tcp [ 4417.198255] Key type .llcrypt unregistered [ 4417.200589] Key type ._llcrypt unregistered [ 4425.457511] Key type ._llcrypt registered [ 4425.458865] Key type .llcrypt registered [ 4425.506490] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 4426.022236] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4426.036724] alg: No test for adler32 (adler32-zlib) [ 4426.917555] Lustre: Lustre: Build Version: 2.17.50_236_gaab9274 [ 4427.018099] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 4428.615169] Key type lgssc registered [ 4429.129286] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4432.486666] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4432.495271] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4433.691637] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4435.194914] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 4438.759255] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4438.933595] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4440.531959] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 4442.702652] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4444.276046] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4446.806518] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4446.930465] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4449.157657] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 4451.395480] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4458.215528] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 4468.705723] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4468.707609] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4468.710215] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4468.715166] Lustre: Skipped 1 previous similar message [ 4472.211117] Lustre: server umount lustre-OST0000 complete [ 4473.824489] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4473.825229] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4473.828744] Lustre: Skipped 1 previous similar message [ 4473.830994] Lustre: Skipped 1 previous similar message [ 4475.808155] Lustre: server umount lustre-MDT0000 complete [ 4477.326292] LustreError: 250573:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773237172 with bad export cookie 11410250272266675700 [ 4477.329780] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4477.334586] LustreError: 250573:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4479.479258] Lustre: server umount lustre-MDT0001 complete [ 4483.841068] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 4485.081810] Key type lgssc unregistered [ 4485.215307] LNet: 253935:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4485.218605] LNetError: 253935:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4485.230574] LNet: Removed LNI 192.168.201.137@tcp [ 4485.578327] Key type .llcrypt unregistered [ 4485.580258] Key type ._llcrypt unregistered [ 4495.580874] Key type ._llcrypt registered [ 4495.582167] Key type .llcrypt registered [ 4495.626235] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 4496.164845] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4496.179428] alg: No test for adler32 (adler32-zlib) [ 4497.053072] Lustre: Lustre: Build Version: 2.17.50_236_gaab9274 [ 4497.159983] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 4498.743115] Key type lgssc registered [ 4499.228161] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4502.424205] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4502.434262] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4503.625556] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4505.243519] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 4508.536724] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4508.685916] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4510.162610] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 4512.266282] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4513.618727] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4516.251867] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4516.397689] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4518.711667] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 4520.893716] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4529.702438] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 4543.967814] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4543.970863] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4543.974929] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4544.993611] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4544.994559] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4544.995219] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4546.588489] Lustre: server umount lustre-OST0000 complete [ 4549.600076] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4549.603304] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4549.609733] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4549.612526] Lustre: Skipped 1 previous similar message [ 4555.232351] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4555.235865] Lustre: Skipped 3 previous similar messages [ 4560.352292] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4560.353738] Lustre: Skipped 1 previous similar message [ 4562.399149] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4562.469136] Lustre: server umount lustre-MDT0000 complete [ 4564.018637] LustreError: 255292:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773237258 with bad export cookie 7380582711116056250 [ 4564.022137] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4564.024432] LustreError: 255292:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4564.167706] Lustre: server umount lustre-MDT0001 complete [ 4568.406539] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 4569.605722] Key type lgssc unregistered [ 4569.735409] LNet: 259282:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4569.738662] LNetError: 259282:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4569.747722] LNet: Removed LNI 192.168.201.137@tcp [ 4570.065638] Key type .llcrypt unregistered [ 4570.067568] Key type ._llcrypt unregistered [ 4577.856632] Key type ._llcrypt registered [ 4577.858843] Key type .llcrypt registered [ 4577.907394] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 4578.294623] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4578.349563] alg: No test for adler32 (adler32-zlib) [ 4579.224344] Lustre: Lustre: Build Version: 2.17.50_236_gaab9274 [ 4579.309367] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 4580.903142] Key type lgssc registered [ 4581.356754] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4584.337234] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4584.346975] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4585.548931] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4586.917246] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 4589.862141] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4589.992203] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4591.569272] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 4593.543876] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4594.859589] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4597.374726] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4597.509386] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4599.658850] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 4601.649251] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4610.851515] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 4616.162462] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4616.163125] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4616.163695] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4616.167660] Lustre: Skipped 1 previous similar message [ 4617.613076] Lustre: server umount lustre-OST0000 complete [ 4620.767834] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4620.770402] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4620.774335] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4620.775945] Lustre: Skipped 1 previous similar message [ 4621.013215] Lustre: server umount lustre-MDT0000 complete [ 4621.280210] LustreError: 260481:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4621.287764] LustreError: 260481:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 3 previous similar messages [ 4622.353131] LustreError: 260527:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773237317 with bad export cookie 7672150007289276656 [ 4622.356401] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4622.357354] LustreError: 260527:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4622.483206] Lustre: server umount lustre-MDT0001 complete [ 4626.142552] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 4627.215424] Key type lgssc unregistered [ 4627.357417] LNet: 263844:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4627.360611] LNetError: 263844:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4627.367299] LNet: Removed LNI 192.168.201.137@tcp [ 4627.682703] Key type .llcrypt unregistered [ 4627.683868] Key type ._llcrypt unregistered [ 4631.559391] Key type ._llcrypt registered [ 4631.561069] Key type .llcrypt registered [ 4631.607023] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 4637.065116] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_hostid [ 4642.868257] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 4643.147171] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4643.155859] alg: No test for adler32 (adler32-zlib) [ 4644.023288] Lustre: Lustre: Build Version: 2.17.50_236_gaab9274 [ 4644.111811] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 4645.695103] Key type lgssc registered [ 4646.107075] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4657.989521] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4657.997297] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4659.086314] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4659.097486] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4659.132820] Lustre: lustre-MDT0000: new disk, initializing [ 4659.157229] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4659.163847] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4660.416291] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 4664.467862] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4664.519784] Lustre: 267317:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4664.555695] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4664.558475] Lustre: Skipped 1 previous similar message [ 4664.610574] Lustre: lustre-MDT0001: new disk, initializing [ 4664.656249] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4664.673569] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4664.678247] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4666.144583] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 4669.400779] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4670.692787] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4673.022583] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4673.136580] Lustre: lustre-OST0000: new disk, initializing [ 4673.139161] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4673.172885] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4675.085621] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 4678.134680] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4678.137779] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4678.170509] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4678.208196] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4679.678773] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4679.764461] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4681.213320] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4681.288209] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4683.234873] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4683.235678] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4683.236276] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4683.238825] LustreError: Skipped 1 previous similar message [ 4683.244338] Lustre: Skipped 1 previous similar message [ 4684.125904] Lustre: server umount lustre-OST0000 complete [ 4688.352854] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4688.353547] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4688.356920] Lustre: Skipped 1 previous similar message [ 4688.361145] Lustre: Skipped 2 previous similar messages [ 4691.683326] Lustre: server umount lustre-MDT0000 complete [ 4692.914400] LustreError: 268094:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773237387 with bad export cookie 2550132857407564257 [ 4692.914901] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4692.919193] LustreError: 268094:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4693.043800] Lustre: server umount lustre-MDT0001 complete [ 4695.974710] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 09:56:29 (1773237389) [ 4698.331179] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4698.542642] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4699.990469] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 4703.091813] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4703.212550] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4704.605693] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 4706.675602] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4707.959765] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4710.531838] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4712.727758] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 4714.888641] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4729.314236] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4729.314595] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4729.318132] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4729.322508] Lustre: Skipped 1 previous similar message [ 4729.326543] Lustre: Skipped 1 previous similar message [ 4734.432980] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4734.434834] Lustre: Skipped 1 previous similar message [ 4734.878330] Lustre: server umount lustre-OST0000 complete [ 4739.040567] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4739.041802] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4739.044332] LustreError: Skipped 1 previous similar message [ 4739.049220] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4742.551679] Lustre: server umount lustre-MDT0000 complete [ 4743.852728] LustreError: 271301:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773237438 with bad export cookie 2550132857407565125 [ 4743.855649] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4743.857975] LustreError: 271301:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4743.996657] Lustre: server umount lustre-MDT0001 complete [ 4748.245120] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 4749.334151] Key type lgssc unregistered [ 4749.471759] LNet: 274526:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4749.474632] LNetError: 274526:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4749.482513] LNet: Removed LNI 192.168.201.137@tcp [ 4749.788433] Key type .llcrypt unregistered [ 4749.790238] Key type ._llcrypt unregistered [ 4759.575865] Key type ._llcrypt registered [ 4759.577127] Key type .llcrypt registered [ 4759.618843] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_hostid [ 4766.276253] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 4766.817866] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4766.831213] alg: No test for adler32 (adler32-zlib) [ 4767.710238] Lustre: Lustre: Build Version: 2.17.50_236_gaab9274 [ 4767.803470] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 4769.383337] Key type lgssc registered [ 4769.832537] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4785.450759] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4785.456470] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4786.552859] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4786.565793] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4786.603423] Lustre: lustre-MDT0000: new disk, initializing [ 4786.630365] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4786.637462] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4788.099226] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 4792.599360] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4792.650663] Lustre: 278286:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4792.672805] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4792.674790] Lustre: Skipped 1 previous similar message [ 4792.716292] Lustre: lustre-MDT0001: new disk, initializing [ 4792.756218] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4792.769001] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4792.774137] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4794.278650] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 4797.525125] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4798.729545] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4801.190386] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4801.298665] Lustre: lustre-OST2710: new disk, initializing [ 4801.300910] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 4801.332176] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 4803.385775] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 4804.026686] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 4804.031859] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 4804.067015] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 4806.590885] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 4809.040944] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4809.104015] Lustre: lustre-OST03e8: new disk, initializing [ 4809.106780] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 4809.138802] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 4811.009281] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 4812.606909] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 4812.610765] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 4812.641871] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 4812.796874] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 4815.403816] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 4815.476272] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4816.710196] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 4816.779980] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4818.129653] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 4818.202413] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4819.587178] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 4819.647614] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4823.009586] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4823.011715] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4823.012967] Lustre: Skipped 1 previous similar message [ 4823.015691] Lustre: Skipped 3 previous similar messages [ 4826.848653] Lustre: server umount lustre-MDT0000 complete [ 4828.118226] LustreError: 278281:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773237522 with bad export cookie 2335933885841898970 [ 4828.121645] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4828.121841] LustreError: 278281:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4828.130365] Lustre: lustre-MDT0001-lwp-OST03e8: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4828.131656] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4828.138735] Lustre: Skipped 3 previous similar messages [ 4828.247422] Lustre: server umount lustre-MDT0001 complete [ 4835.715381] Lustre: server umount lustre-OST2710 complete [ 4836.978616] Lustre: server umount lustre-OST03e8 complete [ 4842.354216] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_hostid [ 4845.322058] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 4856.481315] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4856.569563] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4856.582284] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4856.617960] Lustre: lustre-MDT0000: new disk, initializing [ 4856.637477] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4856.643650] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4857.869828] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 4862.337608] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4862.390220] Lustre: 285997:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4862.500882] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4863.913127] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 4866.951908] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4868.116731] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4870.264707] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4870.371868] Lustre: lustre-OST0000: new disk, initializing [ 4870.373051] Lustre: Skipped 1 previous similar message [ 4870.375592] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4870.377711] Lustre: Skipped 2 previous similar messages [ 4870.406711] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4870.409459] Lustre: Skipped 1 previous similar message [ 4872.311424] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 4872.950114] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4872.952414] Lustre: Skipped 1 previous similar message [ 4872.953956] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4872.985153] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4875.363178] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4876.698878] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4876.772981] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4878.073377] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4878.144789] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4883.425035] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4883.427600] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4883.428045] LustreError: Skipped 1 previous similar message [ 4883.432225] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4883.434879] Lustre: Skipped 2 previous similar messages [ 4888.545900] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4888.548911] Lustre: Skipped 1 previous similar message [ 4893.151128] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4893.212440] Lustre: server umount lustre-OST0000 complete [ 4898.271986] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4898.274747] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4898.278470] Lustre: Skipped 1 previous similar message [ 4898.280240] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4900.630948] Lustre: server umount lustre-MDT0000 complete [ 4901.864180] LustreError: 286774:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773237596 with bad export cookie 2335933885841900636 [ 4901.867814] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4901.869342] LustreError: 286774:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4901.989769] Lustre: server umount lustre-MDT0001 complete [ 4904.758609] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 09:59:58 (1773237598) [ 4905.263237] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 4905.895397] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 09:59:59 (1773237599) [ 4906.910276] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 4915.848897] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4915.944311] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4915.951749] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4916.013413] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4917.327841] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 4920.173993] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4920.209365] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4920.218917] Lustre: 291430:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4921.580354] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 4923.296903] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4924.451723] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4939.745631] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4939.804551] Lustre: Denying initial registration attempt from nid 192.168.201.137@tcp, specified as failover [ 4939.807053] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 4939.809117] LustreError: 293043:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -99 [ 4939.810819] LustreError: 293043:0:(tgt_mount.c:2064:server_put_super()) no obd lustre-OST0000 [ 4939.812203] LustreError: 293043:0:(tgt_mount.c:115:server_deregister_mount()) lustre-OST0000 not registered [ 4939.836135] Lustre: server umount lustre-OST0000 complete [ 4939.837397] LustreError: 293043:0:(super25.c:186:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 4944.352333] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4944.352920] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4944.357801] Lustre: Skipped 2 previous similar messages [ 4944.360491] Lustre: Skipped 1 previous similar message [ 4956.127131] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4956.182865] Lustre: server umount lustre-MDT0000 complete [ 4957.401263] LustreError: 292153:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773237652 with bad export cookie 2335933885841901504 [ 4957.404379] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4957.404530] LustreError: 292153:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4957.519670] Lustre: server umount lustre-MDT0001 complete [ 4961.153863] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 4962.240809] Key type lgssc unregistered [ 4962.371429] LNet: 294145:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4962.375572] LNetError: 294145:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4962.382323] LNet: Removed LNI 192.168.201.137@tcp [ 4962.698513] Key type .llcrypt unregistered [ 4962.700202] Key type ._llcrypt unregistered [ 4966.199424] Key type ._llcrypt registered [ 4966.200632] Key type .llcrypt registered [ 4966.238694] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 4966.580796] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4966.623529] alg: No test for adler32 (adler32-zlib) [ 4967.484270] Lustre: Lustre: Build Version: 2.17.50_236_gaab9274 [ 4967.570717] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 4969.151157] Key type lgssc registered [ 4969.511975] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5003.866146] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 5006.803916] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5006.811302] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5007.906607] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5007.914313] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5007.978501] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5009.239434] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5012.054261] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5012.084640] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5012.094922] Lustre: 296869:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5012.193869] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5013.547813] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5015.297788] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5016.462435] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5031.586847] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5031.662580] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5031.726846] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5033.611270] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5035.470963] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5036.246908] Lustre: server umount lustre-OST0000 complete [ 5038.048516] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5038.051297] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5038.056755] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5041.135504] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5041.135895] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5043.670661] Lustre: server umount lustre-MDT0000 complete [ 5044.937245] LustreError: 298507:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773237739 with bad export cookie 8514257049809496541 [ 5044.940768] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5044.942829] LustreError: 298507:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5045.059980] Lustre: server umount lustre-MDT0001 complete [ 5048.680910] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 5049.723595] Key type lgssc unregistered [ 5049.857331] LNet: 300075:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5049.859498] LNetError: 300075:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5049.868276] LNet: Removed LNI 192.168.201.137@tcp [ 5050.185554] Key type .llcrypt unregistered [ 5050.186598] Key type ._llcrypt unregistered [ 5059.189679] Key type ._llcrypt registered [ 5059.191612] Key type .llcrypt registered [ 5059.233632] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 5059.629381] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5059.676995] alg: No test for adler32 (adler32-zlib) [ 5060.543631] Lustre: Lustre: Build Version: 2.17.50_236_gaab9274 [ 5060.639867] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 5062.231228] Key type lgssc registered [ 5062.640506] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5065.475072] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5065.481471] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5066.648718] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5067.863627] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5070.569510] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5070.678083] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5071.846968] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5073.447791] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5074.577634] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5078.817855] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5078.929529] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5080.625734] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5082.385836] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5096.416979] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5096.418192] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5096.419238] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5096.425103] Lustre: Skipped 2 previous similar messages [ 5097.901062] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.37@tcp (stopping) [ 5097.903182] Lustre: Skipped 2 previous similar messages [ 5101.087494] Lustre: server umount lustre-MDT0000 complete [ 5101.537220] LustreError: 301446:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5101.542189] LustreError: 301446:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 4 previous similar messages [ 5102.401198] LustreError: 301432:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773237797 with bad export cookie 14992399508976096371 [ 5102.404752] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5102.405907] LustreError: 301432:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5103.012767] LustreError: 302887:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5103.020352] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.37@tcp (stopping) [ 5106.656087] LustreError: 301447:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5106.656356] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5106.668796] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5108.590380] Lustre: server umount lustre-MDT0001 complete [ 5110.741769] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 5113.621354] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5128.159687] LustreError: 300763:0:(client.c:1428:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000545a3582 x1859374552211584/t0(0) o250->MGC192.168.201.137@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5128.277887] LustreError: 303054:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5128.283348] LustreError: 303054:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 5128.304443] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5128.323796] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:161) [ 5129.578931] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5132.385447] LustreError: 304727:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5132.391299] LustreError: 304727:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 5132.409055] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5132.540826] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5133.909387] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5135.784894] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5137.701029] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5138.375090] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 5139.621588] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5140.184880] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5143.011230] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5143.012052] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5143.020222] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5143.022060] Lustre: Skipped 3 previous similar messages [ 5147.606734] Lustre: server umount lustre-OST0000 complete [ 5153.249738] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5153.250241] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5153.250915] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5153.254494] Lustre: Skipped 2 previous similar messages [ 5155.030302] Lustre: server umount lustre-MDT0000 complete [ 5156.304370] LustreError: 303067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773237851 with bad export cookie 14992399508976103525 [ 5156.308133] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5156.309073] LustreError: 303067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5156.421718] Lustre: server umount lustre-MDT0001 complete [ 5160.061947] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 5161.119872] Key type lgssc unregistered [ 5161.265375] LNet: 307157:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5161.267705] LNetError: 307157:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5161.273498] LNet: Removed LNI 192.168.201.137@tcp [ 5161.597716] Key type .llcrypt unregistered [ 5161.598849] Key type ._llcrypt unregistered [ 5170.349535] Key type ._llcrypt registered [ 5170.350514] Key type .llcrypt registered [ 5170.383877] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 5170.674500] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5170.720237] alg: No test for adler32 (adler32-zlib) [ 5171.554380] Lustre: Lustre: Build Version: 2.17.50_236_gaab9274 [ 5171.621985] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 5173.199143] Key type lgssc registered [ 5173.568102] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5176.220710] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5176.226831] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5177.389701] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5178.704273] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5181.531387] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5181.655298] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5182.958932] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5184.656471] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5185.930069] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5186.835044] LustreError: 308563:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 5186.837053] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5186.837596] LustreError: 308563:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 5186.837986] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5186.840665] Lustre: Skipped 1 previous similar message [ 5186.844768] LustreError: 308563:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 9, retries 0, failed: rc = -5 [ 5191.968485] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5191.971834] Lustre: Skipped 2 previous similar messages [ 5197.280172] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5197.281991] Lustre: Skipped 1 previous similar message [ 5201.375106] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5201.433073] Lustre: server umount lustre-MDT0000 complete [ 5202.400129] LustreError: 309256:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5202.405214] LustreError: 309256:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 3 previous similar messages [ 5202.734302] LustreError: 308515:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773237897 with bad export cookie 8467918731932212057 [ 5202.738421] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5202.740121] LustreError: 308515:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5202.853692] Lustre: server umount lustre-MDT0001 complete [ 5207.922667] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5208.025019] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5208.037415] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5208.110017] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5209.483636] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5212.453037] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5212.491732] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5212.504201] Lustre: 310894:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5212.602682] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5213.942543] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5215.720256] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5217.010888] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5221.730754] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5221.803298] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 5221.806667] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 5221.809637] LustreError: 312549:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -2 [ 5221.811799] LustreError: 312549:0:(tgt_mount.c:2064:server_put_super()) no obd lustre-OST0000 [ 5221.813649] LustreError: 312549:0:(tgt_mount.c:115:server_deregister_mount()) lustre-OST0000 not registered [ 5221.838102] Lustre: server umount lustre-OST0000 complete [ 5221.839755] LustreError: 312549:0:(super25.c:186:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5226.786799] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5226.857658] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5226.925917] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5228.809832] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5230.598512] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5233.251948] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5233.289861] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5233.317646] Lustre: lustre-OST0001: new disk, initializing [ 5233.319623] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5235.245162] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5235.664595] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5235.669325] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5235.689933] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:193) [ 5235.715275] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5237.047719] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5240.801445] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5240.803680] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5240.807562] Lustre: Skipped 1 previous similar message [ 5240.808081] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5240.812669] Lustre: Skipped 1 previous similar message [ 5244.092810] Lustre: server umount lustre-OST0001 complete [ 5245.925295] LustreError: 313015:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5245.925481] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5245.926070] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5245.926086] LustreError: Skipped 1 previous similar message [ 5245.926889] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5245.931281] LustreError: 313015:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 5245.936724] Lustre: Skipped 1 previous similar message [ 5247.441578] Lustre: server umount lustre-OST0000 complete [ 5251.040942] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5251.046132] Lustre: Skipped 1 previous similar message [ 5254.809246] Lustre: server umount lustre-MDT0000 complete [ 5256.020308] LustreError: 315065:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773237950 with bad export cookie 8467918731932212589 [ 5256.023286] LustreError: 315065:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5256.023572] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5256.129758] Lustre: server umount lustre-MDT0001 complete [ 5259.540666] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 5260.548988] Key type lgssc unregistered [ 5260.688338] LNet: 315798:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5260.690783] LNetError: 315798:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5260.700270] LNet: Removed LNI 192.168.201.137@tcp [ 5260.987285] Key type .llcrypt unregistered [ 5260.988513] Key type ._llcrypt unregistered [ 5291.231792] Key type ._llcrypt registered [ 5291.232935] Key type .llcrypt registered [ 5291.272407] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_hostid [ 5297.470180] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 5297.918418] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5297.924285] alg: No test for adler32 (adler32-zlib) [ 5298.773506] Lustre: Lustre: Build Version: 2.17.50_236_gaab9274 [ 5298.856630] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 5300.431173] Key type lgssc registered [ 5300.830856] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5312.132765] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5312.139414] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5313.252080] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5313.263212] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5313.306045] Lustre: lustre-MDT0000: new disk, initializing [ 5313.336263] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5313.346350] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5314.710055] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5318.872825] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5318.916937] Lustre: 320670:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5318.943997] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5318.945742] Lustre: Skipped 1 previous similar message [ 5318.980784] Lustre: lustre-MDT0001: new disk, initializing [ 5319.019207] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5319.032179] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5319.035754] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5320.298673] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5323.373398] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5324.629116] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5326.790666] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5326.897701] Lustre: lustre-OST0000: new disk, initializing [ 5326.899319] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5326.926734] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5328.566828] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5328.569619] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5328.606138] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5328.783446] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5331.896369] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5333.277112] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5333.352372] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5334.662959] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5334.741436] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5339.106890] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5339.107655] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5339.108077] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5339.112696] Lustre: Skipped 1 previous similar message [ 5339.116845] LustreError: Skipped 1 previous similar message [ 5341.458766] Lustre: server umount lustre-OST0000 complete [ 5344.224593] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5344.225332] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5344.228166] Lustre: Skipped 1 previous similar message [ 5344.233534] Lustre: Skipped 2 previous similar messages [ 5348.829448] Lustre: server umount lustre-MDT0000 complete [ 5350.073261] LustreError: 321451:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773238044 with bad export cookie 6968051425503686342 [ 5350.077318] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5350.078666] LustreError: 321451:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5350.186668] Lustre: server umount lustre-MDT0001 complete [ 5352.942849] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 10:07:26 (1773238046) [ 5355.395655] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 10:07:29 (1773238049) [ 5358.232081] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5358.385292] LustreError: 325015:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5358.391483] LustreError: 325015:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 3 previous similar messages [ 5358.407510] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5359.618284] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5362.355415] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5362.479688] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5363.823573] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5365.668418] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5366.940581] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5369.268163] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5371.325861] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5373.126306] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5392.020818] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 5392.533376] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 5393.112183] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 5393.650749] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 5394.313056] Lustre: Failing over lustre-MDT0000 [ 5394.480641] Lustre: server umount lustre-MDT0000 complete [ 5398.495971] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5398.498038] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5398.501653] LustreError: 325016:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5404.069323] LustreError: 325011:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5404.075198] LustreError: 325011:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 4 previous similar messages [ 5407.351058] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5407.397607] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5407.486211] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5407.488434] Lustre: Skipped 1 previous similar message [ 5407.512396] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5408.841279] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5409.189778] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5412.839747] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5412.848133] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 5412.863057] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5413.997053] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5414.556946] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5416.413513] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 5417.064437] Lustre: Failing over lustre-MDT0000 [ 5417.216039] Lustre: server umount lustre-MDT0000 complete [ 5417.952889] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5417.952968] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5417.957769] Lustre: Skipped 2 previous similar messages [ 5417.958326] LustreError: 325727:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5417.968092] LustreError: 325727:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 5 previous similar messages [ 5423.072091] LustreError: 325016:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5423.076579] LustreError: 325016:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 3 previous similar messages [ 5430.074783] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5430.117849] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5430.207875] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5430.240474] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5431.616511] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5434.790569] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5435.366792] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5435.368914] Lustre: Skipped 2 previous similar messages [ 5435.378742] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 5435.398237] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 5436.572486] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5437.120964] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5440.481462] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5440.482185] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5440.482863] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5440.489322] Lustre: Skipped 3 previous similar messages [ 5445.528360] Lustre: server umount lustre-MDT0000 complete [ 5445.601905] LustreError: 325015:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5445.605756] LustreError: 325015:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 7 previous similar messages [ 5446.751459] LustreError: 325041:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773238141 with bad export cookie 6968051425503689023 [ 5446.754668] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5446.755050] LustreError: 325041:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5446.880227] Lustre: server umount lustre-MDT0001 complete [ 5457.444934] Lustre: server umount lustre-OST0000 complete [ 5462.608258] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5462.779193] LustreError: 330811:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5462.804868] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5464.077196] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5466.887061] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5468.250846] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5469.943803] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5471.009522] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5473.169153] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5475.217595] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5476.087161] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 5476.973582] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5481.440084] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5481.442296] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5481.446570] Lustre: Skipped 1 previous similar message [ 5481.448456] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5481.450247] Lustre: Skipped 2 previous similar messages [ 5484.178725] Lustre: server umount lustre-OST0000 complete [ 5487.585892] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5487.587706] Lustre: Skipped 3 previous similar messages [ 5491.535733] Lustre: server umount lustre-MDT0000 complete [ 5492.648598] LustreError: 330793:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773238187 with bad export cookie 6968051425503689744 [ 5492.651716] LustreError: 330793:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5492.652223] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5492.763964] Lustre: server umount lustre-MDT0001 complete [ 5496.200700] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 5497.156480] Key type lgssc unregistered [ 5497.281416] LNet: 333986:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5497.284846] LNetError: 333986:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5497.293277] LNet: Removed LNI 192.168.201.137@tcp [ 5497.568085] Key type .llcrypt unregistered [ 5497.569093] Key type ._llcrypt unregistered [ 5506.950211] Key type ._llcrypt registered [ 5506.951284] Key type .llcrypt registered [ 5506.987145] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 5507.303613] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5507.357303] alg: No test for adler32 (adler32-zlib) [ 5508.211856] Lustre: Lustre: Build Version: 2.17.50_236_gaab9274 [ 5508.294462] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 5509.871166] Key type lgssc registered [ 5510.225577] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5512.697474] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5512.703269] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5513.870630] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5515.088658] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5517.732852] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5517.855708] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5519.127850] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5520.820834] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5521.971661] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5524.145421] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5524.259675] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5526.123680] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5527.837505] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5538.597692] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 5548.511729] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5548.513775] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5548.517161] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5549.024806] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5549.025785] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5553.246749] Lustre: server umount lustre-MDT0000 complete [ 5554.144923] LustreError: 335463:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5554.156666] LustreError: 335463:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 4 previous similar messages [ 5554.806650] LustreError: 335442:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773238249 with bad export cookie 3919079052118392278 [ 5554.812094] LustreError: 335442:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5554.812326] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5554.931244] Lustre: server umount lustre-MDT0001 complete [ 5566.613162] Lustre: server umount lustre-OST0000 complete [ 5570.930941] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5571.098779] LustreError: 338857:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5571.121600] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5572.434310] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5575.254060] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5575.369094] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5576.672122] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5578.464044] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5579.614402] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5581.758020] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5583.712209] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5585.395916] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5596.194193] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5611.488750] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5611.489574] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5611.491480] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5611.494104] Lustre: Skipped 2 previous similar messages [ 5611.498852] Lustre: Skipped 4 previous similar messages [ 5614.103728] Lustre: server umount lustre-MDT0000 complete [ 5615.293037] LustreError: 338838:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773238310 with bad export cookie 3919079052118393377 [ 5615.296231] LustreError: 338838:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5615.297619] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5615.410422] Lustre: server umount lustre-MDT0001 complete [ 5626.902376] Lustre: server umount lustre-OST0000 complete [ 5631.821172] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5631.968329] LustreError: 342448:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5631.991501] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5631.993316] Lustre: Skipped 1 previous similar message [ 5633.272309] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5636.074131] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5637.415044] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5639.171413] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5640.352573] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5642.587042] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5644.514983] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5646.244434] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5657.063674] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5666.272071] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5666.274565] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5666.278408] Lustre: Skipped 1 previous similar message [ 5666.280281] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5669.454827] Lustre: server umount lustre-OST0000 complete [ 5671.903854] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5671.905970] LustreError: Skipped 1 previous similar message [ 5671.907259] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5671.910635] Lustre: Skipped 1 previous similar message [ 5671.912511] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5671.914353] Lustre: Skipped 2 previous similar messages [ 5676.759581] Lustre: server umount lustre-MDT0000 complete [ 5677.535913] LustreError: 343817:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5677.540285] LustreError: 343817:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 5678.036119] LustreError: 342430:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773238372 with bad export cookie 3919079052118394385 [ 5678.039535] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5678.040294] LustreError: 342430:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5678.164896] Lustre: server umount lustre-MDT0001 complete [ 5681.384922] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 5682.285591] Key type lgssc unregistered [ 5682.413333] LNet: 345650:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5682.415434] LNetError: 345650:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5682.421285] LNet: Removed LNI 192.168.201.137@tcp [ 5682.684644] Key type .llcrypt unregistered [ 5682.685512] Key type ._llcrypt unregistered [ 5691.635619] Key type ._llcrypt registered [ 5691.636545] Key type .llcrypt registered [ 5691.676480] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 5692.003584] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5692.039062] alg: No test for adler32 (adler32-zlib) [ 5692.887488] Lustre: Lustre: Build Version: 2.17.50_236_gaab9274 [ 5692.968952] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 5694.543098] Key type lgssc registered [ 5694.896420] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5697.372406] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5697.378491] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5697.381595] LustreError: 347049:0:(osd_handler.c:8547:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 5697.390281] LustreError: 347049:0:(obd_config.c:880:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 5697.393673] LustreError: 347049:0:(obd_mount.c:217:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 5697.396324] LustreError: 347049:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 5697.399699] LustreError: 347049:0:(super25.c:186:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 5699.863613] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5699.866671] LustreError: 347447:0:(osd_handler.c:8547:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 5699.875968] LustreError: 347447:0:(obd_config.c:880:class_setup()) setup lustre-OST0000-osd failed (-22) [ 5699.878118] LustreError: 347447:0:(obd_mount.c:217:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 5699.880776] LustreError: 347447:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 5699.882881] LustreError: 347447:0:(super25.c:186:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 5706.194094] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 5707.274779] Key type lgssc unregistered [ 5707.404384] LNet: 348430:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5707.406972] LNetError: 348430:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5707.420299] LNet: Removed LNI 192.168.201.137@tcp [ 5707.745662] Key type .llcrypt unregistered [ 5707.747054] Key type ._llcrypt unregistered [ 5713.058170] Key type ._llcrypt registered [ 5713.059467] Key type .llcrypt registered [ 5713.098647] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_hostid [ 5719.179197] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 5719.504118] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5719.535049] alg: No test for adler32 (adler32-zlib) [ 5720.394931] Lustre: Lustre: Build Version: 2.17.50_236_gaab9274 [ 5720.482578] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 5722.063139] Key type lgssc registered [ 5722.422523] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5733.664386] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5733.669922] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5734.763653] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5734.776443] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5734.817127] Lustre: lustre-MDT0000: new disk, initializing [ 5734.842223] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5734.848752] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5736.138791] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5740.243496] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5740.285207] Lustre: 351525:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5740.310166] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5740.313414] Lustre: Skipped 1 previous similar message [ 5740.355354] Lustre: lustre-MDT0001: new disk, initializing [ 5740.389617] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5740.402689] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5740.407315] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5741.673439] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5744.656932] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5745.755082] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5748.004324] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5748.116617] Lustre: lustre-OST0000: new disk, initializing [ 5748.118772] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5748.152833] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5750.076395] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5750.081909] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5750.126924] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5750.240857] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5753.505761] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5754.973553] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5755.051753] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5756.402690] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5756.480941] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5760.481033] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5760.481842] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5760.484069] LustreError: Skipped 1 previous similar message [ 5760.489833] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5760.493578] Lustre: Skipped 1 previous similar message [ 5763.283395] Lustre: server umount lustre-OST0000 complete [ 5765.600668] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5765.601546] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5765.604459] Lustre: Skipped 2 previous similar messages [ 5765.609019] Lustre: Skipped 1 previous similar message [ 5770.649950] Lustre: server umount lustre-MDT0000 complete [ 5771.963392] LustreError: 351516:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773238466 with bad export cookie 10234862575981631265 [ 5771.967392] LustreError: 351516:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5771.967496] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5772.095644] Lustre: server umount lustre-MDT0001 complete [ 5774.874271] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 5775.402550] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 10:14:29 (1773238469) [ 5777.568318] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5777.726463] LustreError: 355574:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5777.732178] LustreError: 355574:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 3 previous similar messages [ 5777.751870] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5779.007465] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5781.835884] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5781.960488] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5783.217371] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5784.987765] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5786.147215] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5788.392337] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5790.400895] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5792.180145] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5794.502112] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5794.571146] Lustre: lustre-OST0001: new disk, initializing [ 5794.573915] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5794.607634] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5794.609580] Lustre: Skipped 1 previous similar message [ 5796.471748] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5797.309068] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5797.312333] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5797.341627] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5799.458055] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5802.276634] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5802.278804] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5802.280816] Lustre: Skipped 1 previous similar message [ 5805.996803] Lustre: lustre-OST0001: Not available for connect from 192.168.201.37@tcp (stopping) [ 5806.001447] Lustre: Skipped 1 previous similar message [ 5807.359200] Lustre: server umount lustre-OST0001 complete [ 5807.588446] LustreError: 357675:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5807.593892] LustreError: 357675:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 5812.707459] LustreError: 357675:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5812.711664] LustreError: 357675:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 5817.827767] LustreError: 358766:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5817.832073] LustreError: 358766:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 5822.431784] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5822.434143] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5822.437991] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5822.948677] LustreError: 358766:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5822.953717] LustreError: 358766:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 5825.423428] Lustre: server umount lustre-OST0000 complete [ 5828.064828] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5828.065903] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5828.067936] LustreError: Skipped 1 previous similar message [ 5828.074556] Lustre: Skipped 1 previous similar message [ 5832.920832] Lustre: server umount lustre-MDT0000 complete [ 5833.184056] LustreError: 355574:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5833.188524] LustreError: 355574:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 5834.106084] LustreError: 355557:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773238528 with bad export cookie 10234862575981632133 [ 5834.109284] LustreError: 355557:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5834.110073] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5840.312151] Lustre: server umount lustre-MDT0001 complete [ 5843.991910] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 5844.981482] Key type lgssc unregistered [ 5845.114394] LNet: 359925:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5845.118050] LNetError: 359925:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5845.131292] LNet: Removed LNI 192.168.201.137@tcp [ 5845.415760] Key type .llcrypt unregistered [ 5845.416663] Key type ._llcrypt unregistered [ 5863.700928] Key type ._llcrypt registered [ 5863.702580] Key type .llcrypt registered [ 5863.739529] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 5864.218608] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5864.230613] alg: No test for adler32 (adler32-zlib) [ 5865.078543] Lustre: Lustre: Build Version: 2.17.50_236_gaab9274 [ 5865.160545] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 5866.743143] Key type lgssc registered [ 5867.083452] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5869.649658] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5869.655315] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5870.745735] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5870.755381] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5870.814596] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5872.100426] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5874.960285] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5874.992556] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5875.002894] Lustre: 362682:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5875.095218] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5876.419146] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5878.228482] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5879.525470] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5880.287854] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5880.289984] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5880.293508] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5884.384168] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5884.384670] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5886.358873] Lustre: server umount lustre-MDT0000 complete [ 5887.624205] LustreError: 363405:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773238582 with bad export cookie 16263458421638712691 [ 5887.627169] LustreError: 363405:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5887.627246] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5887.744672] Lustre: server umount lustre-MDT0001 complete [ 5890.944804] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 10:16:24 (1773238584) [ 5893.104574] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5893.275140] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5894.598093] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5897.457058] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5897.583170] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5898.882864] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5900.574390] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5901.782419] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5904.178923] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5904.257890] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5904.337097] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5906.140026] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5907.874233] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5917.055457] Lustre: Permanently deactivating lustre-OST0000 [ 5917.060787] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 5917.449040] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5918.951935] LustreError: 367464:0:(mgs_llog.c:1942:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [ 5918.955080] LustreError: 367464:0:(mgs_llog.c:1997:mgs_replace_nids()) Only MGS is allowed to be started [ 5918.956989] LustreError: 367464:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 5919.594813] LustreError: 367513:0:(mgs_llog.c:1997:mgs_replace_nids()) Only MGS is allowed to be started [ 5919.597184] LustreError: 367513:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 5926.353130] Lustre: server umount lustre-OST0000 complete [ 5927.274377] LustreError: 367764:0:(mgs_llog.c:1997:mgs_replace_nids()) Only MGS is allowed to be started [ 5927.277941] LustreError: 367764:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 5928.415768] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5928.418411] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5928.422136] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5928.423845] Lustre: Skipped 1 previous similar message [ 5932.000134] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5932.000697] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5937.120214] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5937.121935] Lustre: Skipped 2 previous similar messages [ 5942.239127] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5942.292530] Lustre: server umount lustre-MDT0000 complete [ 5943.458305] LustreError: 364946:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773238638 with bad export cookie 16263458421638713258 [ 5943.461906] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5943.461958] LustreError: 364946:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5954.063549] Lustre: server umount lustre-MDT0001 complete [ 5956.841789] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5958.178045] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5959.136162] LustreError: 368903:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [ 5961.244420] Lustre: 369241:0:(mgs_llog.c:1709:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.201.137@tcp [ 5961.953690] Lustre: server umount MGS complete [ 5965.676436] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5965.825785] LustreError: 369965:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5965.830380] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5965.832908] LustreError: 369965:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 5965.834504] Lustre: Skipped 1 previous similar message [ 5965.853504] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5967.116479] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5969.993567] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5970.107909] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5971.496509] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5973.318053] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5974.461277] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5975.088239] Lustre: Permanently reactivating lustre-OST0000 [ 5975.092851] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 5975.094788] Lustre: Skipped 2 previous similar messages [ 5996.511118] LustreError: 371373:0:(osp_dev.c:694:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [ 5999.529663] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5999.643238] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5999.645377] Lustre: Skipped 1 previous similar message [ 6001.460633] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 6003.171018] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6016.995507] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6016.995726] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6016.997162] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6016.997167] Lustre: Skipped 2 previous similar messages [ 6016.997765] LustreError: Skipped 1 previous similar message [ 6017.002869] Lustre: Skipped 1 previous similar message [ 6021.267260] Lustre: server umount lustre-OST0000 complete [ 6026.720370] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6026.723863] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6028.764262] Lustre: server umount lustre-MDT0000 complete [ 6030.035130] LustreError: 369946:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773238724 with bad export cookie 16263458421638714294 [ 6030.037730] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6030.038412] LustreError: 369946:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6030.149260] Lustre: server umount lustre-MDT0001 complete [ 6033.786434] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 6034.790523] Key type lgssc unregistered [ 6034.899264] LNet: 374139:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6034.901427] LNetError: 374139:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6034.908303] LNet: Removed LNI 192.168.201.137@tcp [ 6035.200336] Key type .llcrypt unregistered [ 6035.201365] Key type ._llcrypt unregistered [ 6040.937472] Key type ._llcrypt registered [ 6040.938475] Key type .llcrypt registered [ 6040.977422] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_hostid [ 6047.222556] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 6047.671162] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6047.682053] alg: No test for adler32 (adler32-zlib) [ 6048.532093] Lustre: Lustre: Build Version: 2.17.50_236_gaab9274 [ 6048.609660] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 6050.191174] Key type lgssc registered [ 6050.560141] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6061.864515] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6061.869905] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6062.968207] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6062.980911] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6063.022919] Lustre: lustre-MDT0000: new disk, initializing [ 6063.051218] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6063.058172] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6064.389477] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 6068.555078] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6068.597815] Lustre: 377232:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6068.622583] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6068.624870] Lustre: Skipped 1 previous similar message [ 6068.667210] Lustre: lustre-MDT0001: new disk, initializing [ 6068.703358] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6068.715815] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6068.719655] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6070.036498] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 6073.174436] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6074.334811] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6076.551797] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6076.666259] Lustre: lustre-OST0000: new disk, initializing [ 6076.668413] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6076.705853] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6078.594129] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 6079.361375] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6079.365986] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6079.404961] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6081.736081] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6083.070057] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6083.153386] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6084.496846] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6084.577442] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6089.696833] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6089.698948] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6089.699410] LustreError: Skipped 1 previous similar message [ 6089.703832] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6089.706606] Lustre: Skipped 1 previous similar message [ 6091.347686] Lustre: server umount lustre-OST0000 complete [ 6094.304129] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6094.306856] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6094.310135] Lustre: Skipped 1 previous similar message [ 6094.311744] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6098.776302] Lustre: server umount lustre-MDT0000 complete [ 6100.036293] LustreError: 378010:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773238794 with bad export cookie 14627334381375181633 [ 6100.038519] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6100.040253] LustreError: 378010:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6100.166537] Lustre: server umount lustre-MDT0001 complete [ 6102.896202] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [ 6103.421285] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 10:19:57 (1773238797) [ 6105.629580] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6105.783755] LustreError: 381280:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6105.792249] LustreError: 381280:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 3 previous similar messages [ 6105.808659] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6107.090840] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 6109.897308] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6110.030060] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6111.331739] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 6113.130385] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6114.284945] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6116.549409] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6118.400793] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 6120.145424] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6120.760997] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [ 6136.810216] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [ 6140.384087] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6140.387260] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6140.391177] Lustre: Skipped 1 previous similar message [ 6140.392985] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6140.394954] Lustre: Skipped 1 previous similar message [ 6142.432770] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6142.433599] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6142.435162] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6142.444401] Lustre: Skipped 1 previous similar message [ 6144.266668] Lustre: server umount lustre-OST0000 complete [ 6147.552830] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6147.553950] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6147.554867] Lustre: Skipped 1 previous similar message [ 6147.558495] Lustre: Skipped 1 previous similar message [ 6151.643819] Lustre: server umount lustre-MDT0000 complete [ 6152.671980] LustreError: 381994:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6152.676201] LustreError: 381994:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 6152.827126] LustreError: 382001:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773238847 with bad export cookie 14627334381375182501 [ 6152.830384] LustreError: 382001:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6152.830588] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6152.945778] Lustre: server umount lustre-MDT0001 complete [ 6156.313331] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 6157.311553] Key type lgssc unregistered [ 6157.442368] LNet: 384674:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6157.444337] LNetError: 384674:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6157.455297] LNet: Removed LNI 192.168.201.137@tcp [ 6157.748639] Key type .llcrypt unregistered [ 6157.749821] Key type ._llcrypt unregistered [ 6165.400049] Key type ._llcrypt registered [ 6165.401694] Key type .llcrypt registered [ 6165.442851] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 6165.552359] Key type .llcrypt unregistered [ 6165.553306] Key type ._llcrypt unregistered [ 6172.163186] Key type ._llcrypt registered [ 6172.164255] Key type .llcrypt registered [ 6172.205201] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 6172.638252] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6172.666082] alg: No test for adler32 (adler32-zlib) [ 6173.532667] Lustre: Lustre: Build Version: 2.17.50_236_gaab9274 [ 6173.624585] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 6175.207150] Key type lgssc registered [ 6175.604439] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6178.255069] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6178.260615] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6179.440113] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6180.766018] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 6183.585653] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6183.701951] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6185.471749] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 6187.136725] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6189.027120] LustreError: 387553:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6189.032055] LustreError: 387553:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 4 previous similar messages [ 6189.450770] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6189.594908] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6190.633957] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6190.942367] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 6194.656188] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6194.659402] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6194.666444] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6195.685879] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6195.686840] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6195.688324] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6199.818200] Lustre: server umount lustre-OST0000 complete [ 6204.895769] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6204.897052] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6204.898162] LustreError: Skipped 1 previous similar message [ 6204.901561] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6204.903734] Lustre: Skipped 1 previous similar message [ 6207.121820] Lustre: server umount lustre-MDT0000 complete [ 6208.364709] LustreError: 388440:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773238903 with bad export cookie 3302120324979776590 [ 6208.369103] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6208.495973] Lustre: server umount lustre-MDT0001 complete [ 6211.962489] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 6212.969750] Key type lgssc unregistered [ 6213.088398] LNet: 389812:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6213.090734] LNetError: 389812:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6213.101259] LNet: Removed LNI 192.168.201.137@tcp [ 6213.389622] Key type .llcrypt unregistered [ 6213.390719] Key type ._llcrypt unregistered [ 6222.166700] Key type ._llcrypt registered [ 6222.168530] Key type .llcrypt registered [ 6222.204865] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 6222.582060] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6222.621993] alg: No test for adler32 (adler32-zlib) [ 6223.483287] Lustre: Lustre: Build Version: 2.17.50_236_gaab9274 [ 6223.560545] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 6225.135152] Key type lgssc registered [ 6225.462201] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6227.896047] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6227.902508] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6243.359367] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 6248.479578] LustreError: 391161:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.137@tcp: failed processing log, type 4: rc = -110 [ 6275.103151] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6276.903493] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 6278.656324] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6280.909672] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6281.087321] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6281.108767] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 6282.339536] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 6285.115662] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6285.261407] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6286.639736] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 6288.530707] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6289.719847] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6294.498904] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6294.499857] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6294.504031] Lustre: Skipped 1 previous similar message [ 6294.506927] Lustre: Skipped 1 previous similar message [ 6298.066435] Lustre: server umount lustre-OST0000 complete [ 6299.616422] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6299.617088] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6299.619776] Lustre: Skipped 1 previous similar message [ 6304.736096] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6304.737788] Lustre: Skipped 2 previous similar messages [ 6305.428720] Lustre: server umount lustre-MDT0000 complete [ 6306.632413] LustreError: 391168:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773239001 with bad export cookie 8666774314939372073 [ 6306.635552] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6306.638364] LustreError: 391168:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6306.764512] Lustre: server umount lustre-MDT0001 complete [ 6310.461235] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 6311.545605] Key type lgssc unregistered [ 6311.696357] LNet: 394376:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6311.698543] LNetError: 394376:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6311.706316] LNet: Removed LNI 192.168.201.137@tcp [ 6311.982080] Key type .llcrypt unregistered [ 6311.983100] Key type ._llcrypt unregistered [ 6321.355376] Key type ._llcrypt registered [ 6321.356412] Key type .llcrypt registered [ 6321.405508] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 6321.744829] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6321.752446] alg: No test for adler32 (adler32-zlib) [ 6322.628934] Lustre: Lustre: Build Version: 2.17.50_236_gaab9274 [ 6322.727907] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 6324.319207] Key type lgssc registered [ 6324.697676] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6327.389781] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6327.395603] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6328.561822] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6329.926089] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 6332.731045] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6332.859690] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6334.111967] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 6335.794674] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6336.992445] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6339.168318] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6339.282693] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6341.114571] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 6342.946806] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6353.637727] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 6363.564301] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.37@tcp (stopping) [ 6363.616155] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6363.619536] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6364.128649] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6364.130485] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6364.130733] Lustre: Skipped 1 previous similar message [ 6365.470059] Lustre: server umount lustre-MDT0000 complete [ 6366.248447] LustreError: 395733:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.201.37@tcp arrived at 1773239061 with bad export cookie 2785281703581618939 [ 6366.252298] LustreError: 395733:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6366.893259] LustreError: 397365:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773239061 with bad export cookie 2785281703581618316 [ 6366.897045] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6366.899630] LustreError: 397365:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6368.223986] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6368.226453] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6368.230566] Lustre: Skipped 1 previous similar message [ 6368.232751] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6368.236078] Lustre: Skipped 1 previous similar message [ 6369.247899] LustreError: 395751:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6369.252533] LustreError: 395751:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 3 previous similar messages [ 6371.296224] LustreError: 396460:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6371.296766] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6371.303535] LustreError: 396460:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 6371.308141] Lustre: Skipped 1 previous similar message [ 6373.019309] Lustre: server umount lustre-OST0000 complete [ 6375.396761] Lustre: server umount lustre-MDT0001 complete [ 6379.020127] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 6380.067913] Key type lgssc unregistered [ 6380.208373] LNet: 399101:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6380.210672] LNetError: 399101:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6380.217262] LNet: Removed LNI 192.168.201.137@tcp [ 6380.511113] Key type .llcrypt unregistered [ 6380.512728] Key type ._llcrypt unregistered [ 6389.864001] Key type ._llcrypt registered [ 6389.865444] Key type .llcrypt registered [ 6389.908703] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 6390.281357] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6390.330884] alg: No test for adler32 (adler32-zlib) [ 6391.203786] Lustre: Lustre: Build Version: 2.17.50_236_gaab9274 [ 6391.302873] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 6392.895122] Key type lgssc registered [ 6393.252989] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6395.620581] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6395.624977] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6396.768570] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6398.002607] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 6400.638762] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6400.749901] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6401.913411] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 6403.477767] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6404.514825] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6406.565522] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6406.661093] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6407.719938] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 6408.316745] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 6409.894605] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6415.841520] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6415.845254] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6415.846591] Lustre: Skipped 1 previous similar message [ 6417.805721] Lustre: server umount lustre-MDT0001 complete [ 6420.960260] LustreError: 401199:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6420.961237] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6420.966086] LustreError: 401199:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 6420.971625] Lustre: Skipped 1 previous similar message [ 6420.976284] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6420.977817] Lustre: Skipped 1 previous similar message [ 6425.425779] Lustre: server umount lustre-OST0000 complete [ 6426.080785] LustreError: 400478:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6426.795579] Lustre: server umount lustre-MDT0000 complete [ 6431.053480] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 6432.084101] Key type lgssc unregistered [ 6432.207475] LNet: 403803:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6432.209707] LNetError: 403803:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6432.218326] LNet: Removed LNI 192.168.201.137@tcp [ 6432.529949] Key type .llcrypt unregistered [ 6432.531032] Key type ._llcrypt unregistered [ 6440.661341] Key type ._llcrypt registered [ 6440.662718] Key type .llcrypt registered [ 6440.703217] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_hostid [ 6446.758210] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 6447.118515] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6447.167380] alg: No test for adler32 (adler32-zlib) [ 6448.021354] Lustre: Lustre: Build Version: 2.17.50_236_gaab9274 [ 6448.109590] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 6449.687167] Key type lgssc registered [ 6450.076127] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6462.255910] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 6467.286908] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6467.292361] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6468.384355] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6468.395903] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6468.435779] Lustre: lustre-MDT0000: new disk, initializing [ 6468.462274] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6468.468693] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6469.790577] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 6475.535450] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6475.633403] Lustre: lustre-OST0000: new disk, initializing [ 6475.635272] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6475.637020] Lustre: Skipped 1 previous similar message [ 6475.659033] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6477.442277] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 6477.445395] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 6477.454177] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 6477.503719] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 6480.528162] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6485.910451] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6485.953270] Lustre: 407735:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6485.980450] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6486.018498] Lustre: lustre-MDT0001: new disk, initializing [ 6486.062203] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6486.077284] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 6486.080608] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 6487.334259] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 6490.795608] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 6490.866992] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 6492.140252] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 6496.287912] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 4 sec [ 6499.245122] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.37@tcp (stopping) [ 6500.320747] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 6500.322333] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6500.328203] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6500.330189] Lustre: Skipped 1 previous similar message [ 6504.357201] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.37@tcp (stopping) [ 6509.476786] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.37@tcp (stopping) [ 6509.479237] Lustre: Skipped 2 previous similar messages [ 6512.095112] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6512.150225] Lustre: server umount lustre-MDT0001 complete [ 6520.801187] LustreError: 407742:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6520.806635] LustreError: 407742:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 4 previous similar messages [ 6525.920737] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6525.920779] LustreError: 407748:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6525.924157] Lustre: Skipped 1 previous similar message [ 6525.930318] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6525.931934] Lustre: Skipped 2 previous similar messages [ 6530.513874] Lustre: server umount lustre-OST0000 complete [ 6531.041061] LustreError: 407747:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6546.399070] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6546.485499] Lustre: server umount lustre-MDT0000 complete [ 6550.785817] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 6551.783377] Key type lgssc unregistered [ 6551.906383] LNet: 412333:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6551.909593] LNetError: 412333:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6551.920353] LNet: Removed LNI 192.168.201.137@tcp [ 6552.210360] Key type .llcrypt unregistered [ 6552.212190] Key type ._llcrypt unregistered [ 6564.312532] Key type ._llcrypt registered [ 6564.313484] Key type .llcrypt registered [ 6564.348902] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 6564.447243] Key type .llcrypt unregistered [ 6564.448247] Key type ._llcrypt unregistered [ 6570.676601] Key type ._llcrypt registered [ 6570.677836] Key type .llcrypt registered [ 6570.712782] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 6571.117189] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6571.126577] alg: No test for adler32 (adler32-zlib) [ 6571.979484] Lustre: Lustre: Build Version: 2.17.50_236_gaab9274 [ 6572.056559] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 6573.631178] Key type lgssc registered [ 6573.980913] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6582.271935] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6582.276888] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6583.361140] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6583.371415] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6583.406963] Lustre: lustre-MDT0000: new disk, initializing [ 6583.430975] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6583.437717] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6584.681986] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 6588.707850] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6588.748380] Lustre: 415648:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6588.770218] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6588.771923] Lustre: Skipped 1 previous similar message [ 6588.806055] Lustre: lustre-MDT0001: new disk, initializing [ 6588.836462] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6588.848024] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6588.851496] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6590.068389] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 6593.044935] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6594.171391] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6596.330831] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6596.433270] Lustre: lustre-OST0000: new disk, initializing [ 6596.435237] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6596.461257] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6598.292800] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 6600.058368] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6600.062701] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6600.105812] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6601.357220] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6604.256128] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6604.258365] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6604.262537] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6605.281762] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6605.283034] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6605.289250] Lustre: Skipped 2 previous similar messages [ 6609.816332] Lustre: server umount lustre-MDT0000 complete [ 6611.028163] LustreError: 416424:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773239305 with bad export cookie 8784670509060627079 [ 6611.031401] LustreError: 416424:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6611.032895] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6611.140781] Lustre: server umount lustre-MDT0001 complete [ 6622.623158] Lustre: server umount lustre-OST0000 complete [ 6625.879585] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 10:28:39 (1773239319) [ 6628.372287] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6628.514984] LustreError: 419301:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6628.523457] LustreError: 419301:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 4 previous similar messages [ 6628.537474] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6629.779406] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 6632.569070] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6632.689456] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6633.972729] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 6635.692827] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6636.864385] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6652.130024] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6652.203682] Lustre: Found index 0 for lustre-OST0000, updating log [ 6652.263140] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6653.287399] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [ 6654.111683] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 6655.876476] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6658.148180] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6658.148262] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6658.153489] Lustre: Skipped 1 previous similar message [ 6659.217102] Lustre: server umount lustre-OST0000 complete [ 6671.781739] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6671.787577] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6671.789338] Lustre: Skipped 3 previous similar messages [ 6675.872554] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6675.874648] Lustre: Skipped 3 previous similar messages [ 6686.175168] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6686.176475] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6686.181312] Lustre: Skipped 3 previous similar messages [ 6686.254916] Lustre: server umount lustre-MDT0000 complete [ 6687.451172] LustreError: 419283:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773239382 with bad export cookie 8784670509060628458 [ 6687.453824] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6687.455425] LustreError: 419283:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6687.560653] Lustre: server umount lustre-MDT0001 complete [ 6690.301485] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 10:29:44 (1773239384) [ 6721.567261] LNet: 1 local NIs in recovery (showing 1): 192.168.201.137@tcp [ 7269.631329] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7269.805867] LustreError: 425368:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7269.835234] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7271.092386] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 7273.878163] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7275.260177] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 7277.027112] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7278.189617] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7293.409800] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7293.489433] Lustre: Found index 0 for lustre-OST0000, updating log [ 7295.334172] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 7296.678079] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:65) [ 7297.015156] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7300.770032] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7300.773825] Lustre: Skipped 1 previous similar message [ 7300.775510] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7303.816850] Lustre: server umount lustre-OST0000 complete [ 7306.016746] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7306.019849] Lustre: Skipped 1 previous similar message [ 7311.123578] Lustre: server umount lustre-MDT0000 complete [ 7311.138170] LustreError: 425368:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7311.145381] LustreError: 425368:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 7312.160114] LustreError: 425369:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7312.164906] LustreError: 425369:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 7312.297285] LustreError: 425349:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773240007 with bad export cookie 8784670509060629515 [ 7312.299846] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7312.301296] LustreError: 425349:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7312.420614] Lustre: server umount lustre-MDT0001 complete [ 7322.655330] LNet: 1 local NIs in recovery (showing 1): 192.168.201.137@tcp [ 7329.310558] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_hostid [ 7332.222923] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 7343.775110] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 10:40:37 (1773240037) [ 7349.897684] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 7351.028266] Key type lgssc unregistered [ 7355.232097] LNet: 431627:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7355.237280] LNetError: 431627:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7423.845337] LNet: Removed LNI 192.168.201.137@tcp [ 7424.138134] Key type .llcrypt unregistered [ 7424.139301] Key type ._llcrypt unregistered [ 7430.730158] Key type ._llcrypt registered [ 7430.731123] Key type .llcrypt registered [ 7430.765455] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 7431.214103] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7431.222374] alg: No test for adler32 (adler32-zlib) [ 7432.081421] Lustre: Lustre: Build Version: 2.17.50_236_gaab9274 [ 7432.165627] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 7433.743199] Key type lgssc registered [ 7434.116379] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7435.588156] LNet: Added LNI 192.168.201.137@tcp100 [8/256/0/180] [ 7435.891215] LNet: Added LNI 192.168.201.137@tcp101 [8/256/0/180] [ 7436.191987] LNet: Added LNI 192.168.201.137@tcp102 [8/256/0/180] [ 7436.499310] LNet: Added LNI 192.168.201.137@tcp103 [8/256/0/180] [ 7436.816928] LNet: Added LNI 192.168.201.137@tcp104 [8/256/0/180] [ 7437.121413] LNet: Added LNI 192.168.201.137@tcp105 [8/256/0/180] [ 7437.431965] LNet: Added LNI 192.168.201.137@tcp106 [8/256/0/180] [ 7437.745278] LNet: Added LNI 192.168.201.137@tcp107 [8/256/0/180] [ 7438.044711] LNet: Added LNI 192.168.201.137@tcp108 [8/256/0/180] [ 7438.352352] LNet: Added LNI 192.168.201.137@tcp109 [8/256/0/180] [ 7438.648730] LNet: Added LNI 192.168.201.137@tcp110 [8/256/0/180] [ 7438.944947] LNet: Added LNI 192.168.201.137@tcp111 [8/256/0/180] [ 7439.240031] LNet: Added LNI 192.168.201.137@tcp112 [8/256/0/180] [ 7439.533646] LNet: Added LNI 192.168.201.137@tcp113 [8/256/0/180] [ 7439.822422] LNet: Added LNI 192.168.201.137@tcp114 [8/256/0/180] [ 7440.136358] LNet: Added LNI 192.168.201.137@tcp115 [8/256/0/180] [ 7440.425685] LNet: Added LNI 192.168.201.137@tcp116 [8/256/0/180] [ 7440.707234] LNet: Added LNI 192.168.201.137@tcp117 [8/256/0/180] [ 7440.999108] LNet: Added LNI 192.168.201.137@tcp118 [8/256/0/180] [ 7441.296883] LNet: Added LNI 192.168.201.137@tcp119 [8/256/0/180] [ 7441.599525] LNet: Added LNI 192.168.201.137@tcp120 [8/256/0/180] [ 7441.894720] LNet: Added LNI 192.168.201.137@tcp121 [8/256/0/180] [ 7442.210642] LNet: Added LNI 192.168.201.137@tcp122 [8/256/0/180] [ 7442.511973] LNet: Added LNI 192.168.201.137@tcp123 [8/256/0/180] [ 7442.800490] LNet: Added LNI 192.168.201.137@tcp124 [8/256/0/180] [ 7443.098370] LNet: Added LNI 192.168.201.137@tcp125 [8/256/0/180] [ 7443.408018] LNet: Added LNI 192.168.201.137@tcp126 [8/256/0/180] [ 7443.703410] LNet: Added LNI 192.168.201.137@tcp127 [8/256/0/180] [ 7444.011332] LNet: Added LNI 192.168.201.137@tcp128 [8/256/0/180] [ 7444.288950] LNet: Added LNI 192.168.201.137@tcp129 [8/256/0/180] [ 7444.569683] LNet: Added LNI 192.168.201.137@tcp130 [8/256/0/180] [ 7444.876979] LNet: Added LNI 192.168.201.137@tcp131 [8/256/0/180] [ 7445.175486] LNet: Added LNI 192.168.201.137@tcp132 [8/256/0/180] [ 7445.461769] LNet: Added LNI 192.168.201.137@tcp133 [8/256/0/180] [ 7445.745357] LNet: Added LNI 192.168.201.137@tcp134 [8/256/0/180] [ 7446.022199] LNet: Added LNI 192.168.201.137@tcp135 [8/256/0/180] [ 7448.208968] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7448.217063] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7449.258758] Lustre: 434607:0:(lustre_peer.c:105:class_add_uuid()) 192.168.201.137@tcp: can't add NID 192.168.201.137@tcp130: rc = -75 [ 7449.318119] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7449.327918] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7449.361570] Lustre: lustre-MDT0000: new disk, initializing [ 7449.390708] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7449.397168] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7450.620268] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 7454.745974] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7454.799990] Lustre: 434624:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7454.826804] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7454.828495] Lustre: Skipped 1 previous similar message [ 7454.866230] Lustre: lustre-MDT0001: new disk, initializing [ 7454.877548] Lustre: 435409:0:(lustre_peer.c:105:class_add_uuid()) 192.168.201.137@tcp: can't add NID 192.168.201.137@tcp130: rc = -75 [ 7454.879713] Lustre: 435409:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [ 7454.925523] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7454.938913] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7454.943385] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7456.195938] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 7459.088256] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7460.204673] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7462.302629] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7462.406241] Lustre: lustre-OST0000: new disk, initializing [ 7462.407979] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7462.442469] Lustre: 436375:0:(lustre_peer.c:105:class_add_uuid()) 192.168.201.137@tcp: can't add NID 192.168.201.137@tcp130: rc = -75 [ 7462.444783] Lustre: 436375:0:(lustre_peer.c:105:class_add_uuid()) Skipped 17 previous similar messages [ 7462.460226] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7464.277561] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 7465.069494] Lustre: 436809:0:(lustre_peer.c:105:class_add_uuid()) 192.168.201.137@tcp: can't add NID 192.168.201.137@tcp130: rc = -75 [ 7465.072809] Lustre: 436809:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [ 7465.088377] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7465.091564] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7465.129406] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7467.393294] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7470.561369] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7470.562984] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7470.564016] LustreError: Skipped 1 previous similar message [ 7470.573021] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7470.576854] Lustre: Skipped 1 previous similar message [ 7475.681288] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7475.683220] Lustre: Skipped 1 previous similar message [ 7480.801832] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7480.803424] Lustre: Skipped 1 previous similar message [ 7484.383187] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7484.438331] Lustre: server umount lustre-OST0000 complete [ 7485.920635] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7485.920823] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7485.921403] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7485.928113] Lustre: Skipped 2 previous similar messages [ 7491.040113] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7491.042234] Lustre: Skipped 2 previous similar messages [ 7500.255089] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7500.320174] Lustre: server umount lustre-MDT0000 complete [ 7501.559573] LustreError: 434617:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773240196 with bad export cookie 11154879565032611281 [ 7501.563282] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7501.563869] LustreError: 434617:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7501.680830] Lustre: server umount lustre-MDT0001 complete [ 7505.195982] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 7506.171475] Key type lgssc unregistered [ 7506.326865] LNet: 438155:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7506.330227] LNetError: 438155:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7506.332708] LNet: Removed LNI 192.168.201.137@tcp [ 7507.359167] LNet: Removed LNI 192.168.201.137@tcp100 [ 7507.360723] LNetError: 438155:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7507.362622] LNetError: 438155:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 7507.364975] LNet: Removed LNI 192.168.201.137@tcp101 [ 7508.383143] LNet: Removed LNI 192.168.201.137@tcp102 [ 7508.384718] LNetError: 438155:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7508.387524] LNetError: 438155:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 7508.389703] LNet: Removed LNI 192.168.201.137@tcp103 [ 7509.407138] LNet: Removed LNI 192.168.201.137@tcp104 [ 7510.431123] LNet: Removed LNI 192.168.201.137@tcp105 [ 7510.432409] LNetError: 438155:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7510.435481] LNetError: 438155:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 7510.437605] LNet: Removed LNI 192.168.201.137@tcp106 [ 7511.455119] LNet: Removed LNI 192.168.201.137@tcp107 [ 7512.479123] LNet: Removed LNI 192.168.201.137@tcp108 [ 7513.503139] LNet: Removed LNI 192.168.201.137@tcp109 [ 7514.527103] LNet: Removed LNI 192.168.201.137@tcp110 [ 7514.528260] LNetError: 438155:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7514.530230] LNetError: 438155:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 7514.532362] LNet: Removed LNI 192.168.201.137@tcp111 [ 7515.551093] LNet: Removed LNI 192.168.201.137@tcp112 [ 7516.575143] LNet: Removed LNI 192.168.201.137@tcp113 [ 7516.577257] LNet: Removed LNI 192.168.201.137@tcp114 [ 7516.578239] LNet: Removed LNI 192.168.201.137@tcp115 [ 7516.579220] LNet: Removed LNI 192.168.201.137@tcp116 [ 7516.580248] LNet: Removed LNI 192.168.201.137@tcp117 [ 7516.581711] LNet: Removed LNI 192.168.201.137@tcp118 [ 7516.582771] LNet: Removed LNI 192.168.201.137@tcp119 [ 7516.584235] LNet: Removed LNI 192.168.201.137@tcp120 [ 7516.585807] LNet: Removed LNI 192.168.201.137@tcp121 [ 7516.587114] LNet: Removed LNI 192.168.201.137@tcp122 [ 7516.588714] LNet: Removed LNI 192.168.201.137@tcp123 [ 7516.589705] LNet: Removed LNI 192.168.201.137@tcp124 [ 7516.591198] LNet: Removed LNI 192.168.201.137@tcp125 [ 7516.592205] LNet: Removed LNI 192.168.201.137@tcp126 [ 7516.593375] LNet: Removed LNI 192.168.201.137@tcp127 [ 7516.594490] LNet: Removed LNI 192.168.201.137@tcp128 [ 7516.596378] LNet: Removed LNI 192.168.201.137@tcp129 [ 7516.597505] LNet: Removed LNI 192.168.201.137@tcp130 [ 7516.598479] LNet: Removed LNI 192.168.201.137@tcp131 [ 7516.599549] LNet: Removed LNI 192.168.201.137@tcp132 [ 7516.600985] LNet: Removed LNI 192.168.201.137@tcp133 [ 7516.602495] LNet: Removed LNI 192.168.201.137@tcp134 [ 7516.609583] LNet: Removed LNI 192.168.201.137@tcp135 [ 7516.866831] Key type .llcrypt unregistered [ 7516.868055] Key type ._llcrypt unregistered [ 7523.471896] Key type ._llcrypt registered [ 7523.473347] Key type .llcrypt registered [ 7523.509625] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_hostid [ 7529.293569] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 7529.638858] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7529.694099] alg: No test for adler32 (adler32-zlib) [ 7530.543406] Lustre: Lustre: Build Version: 2.17.50_236_gaab9274 [ 7530.627440] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 7532.207226] Key type lgssc registered [ 7532.554543] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7543.084772] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 10:43:57 (1773240237) [ 7549.257701] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 7550.259407] Key type lgssc unregistered [ 7550.375363] LNet: 442159:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7550.378230] LNetError: 442159:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7550.389303] LNet: Removed LNI 192.168.201.137@tcp [ 7550.674435] Key type .llcrypt unregistered [ 7550.675539] Key type ._llcrypt unregistered [ 7557.044042] Key type ._llcrypt registered [ 7557.045512] Key type .llcrypt registered [ 7557.084427] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 7557.527161] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7557.536029] alg: No test for adler32 (adler32-zlib) [ 7558.384801] Lustre: Lustre: Build Version: 2.17.50_236_gaab9274 [ 7558.462984] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 7560.039117] Key type lgssc registered [ 7560.395266] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7563.454462] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7563.461777] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7564.549822] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7564.560847] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7564.595936] Lustre: lustre-MDT0000: new disk, initializing [ 7564.619379] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7564.625873] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7565.908265] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 7569.943421] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7569.984138] Lustre: 443331:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7570.008745] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7570.012321] Lustre: Skipped 1 previous similar message [ 7570.049869] Lustre: lustre-MDT0001: new disk, initializing [ 7570.082704] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7570.096345] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7570.100696] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7571.352720] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 7574.288302] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7575.375424] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7577.455924] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7577.546748] Lustre: lustre-OST0000: new disk, initializing [ 7577.549031] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7577.574613] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7579.298227] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 7579.828901] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7579.832285] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7579.863211] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7582.252887] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7583.299226] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing lnet_if_list [ 7584.824123] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [ 7584.834456] LNet: Added LNI 192.168.201.137@tcp10 [8/256/0/180] [ 7586.447309] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [ 7586.457699] LNet: Added LNI 192.168.201.137@tcp11 [8/256/0/180] [ 7587.623695] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [ 7587.634319] LNet: Added LNI 192.168.201.137@tcp12 [8/256/0/180] [ 7588.852423] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [ 7588.864514] LNet: Added LNI 192.168.201.137@tcp13 [8/256/0/180] [ 7590.128369] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [ 7590.138777] LNet: Added LNI 192.168.201.137@tcp14 [8/256/0/180] [ 7591.464201] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [ 7591.477977] LNet: Added LNI 192.168.201.137@tcp15 [8/256/0/180] [ 7592.823250] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [ 7592.834900] LNet: Added LNI 192.168.201.137@tcp16 [8/256/0/180] [ 7594.242706] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [ 7594.253334] LNet: Added LNI 192.168.201.137@tcp17 [8/256/0/180] [ 7595.634410] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [ 7595.643770] LNet: Added LNI 192.168.201.137@tcp18 [8/256/0/180] [ 7597.095470] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [ 7597.106640] LNet: Added LNI 192.168.201.137@tcp19 [8/256/0/180] [ 7598.582554] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [ 7598.593470] LNet: Added LNI 192.168.201.137@tcp20 [8/256/0/180] [ 7600.046107] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [ 7600.057270] LNet: Added LNI 192.168.201.137@tcp21 [8/256/0/180] [ 7601.624597] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [ 7601.634982] LNet: Added LNI 192.168.201.137@tcp22 [8/256/0/180] [ 7603.249095] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [ 7603.258554] LNet: Added LNI 192.168.201.137@tcp23 [8/256/0/180] [ 7604.880524] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [ 7604.891106] LNet: Added LNI 192.168.201.137@tcp24 [8/256/0/180] [ 7606.591758] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [ 7606.604268] LNet: Added LNI 192.168.201.137@tcp25 [8/256/0/180] [ 7608.396243] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [ 7608.409073] LNet: Added LNI 192.168.201.137@tcp26 [8/256/0/180] [ 7610.375598] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [ 7610.389267] LNet: Added LNI 192.168.201.137@tcp27 [8/256/0/180] [ 7612.278758] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [ 7612.287676] LNet: Added LNI 192.168.201.137@tcp28 [8/256/0/180] [ 7614.150961] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [ 7614.165097] LNet: Added LNI 192.168.201.137@tcp29 [8/256/0/180] [ 7616.080600] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [ 7616.092070] LNet: Added LNI 192.168.201.137@tcp30 [8/256/0/180] [ 7618.101990] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [ 7618.111708] LNet: Added LNI 192.168.201.137@tcp31 [8/256/0/180] [ 7620.029533] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [ 7620.041900] LNet: Added LNI 192.168.201.137@tcp32 [8/256/0/180] [ 7621.838445] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [ 7621.848436] LNet: Added LNI 192.168.201.137@tcp33 [8/256/0/180] [ 7623.739412] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [ 7623.750406] LNet: Added LNI 192.168.201.137@tcp34 [8/256/0/180] [ 7625.593904] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [ 7625.603451] LNet: Added LNI 192.168.201.137@tcp35 [8/256/0/180] [ 7627.717985] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [ 7627.729411] LNet: Added LNI 192.168.201.137@tcp36 [8/256/0/180] [ 7629.767874] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [ 7629.778800] LNet: Added LNI 192.168.201.137@tcp37 [8/256/0/180] [ 7631.765701] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [ 7631.779328] LNet: Added LNI 192.168.201.137@tcp38 [8/256/0/180] [ 7633.911322] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [ 7633.921398] LNet: Added LNI 192.168.201.137@tcp39 [8/256/0/180] [ 7635.966380] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [ 7635.978227] LNet: Added LNI 192.168.201.137@tcp40 [8/256/0/180] [ 7638.164809] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [ 7638.176232] LNet: Added LNI 192.168.201.137@tcp41 [8/256/0/180] [ 7640.273509] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [ 7640.282737] LNet: Added LNI 192.168.201.137@tcp42 [8/256/0/180] [ 7642.420663] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [ 7642.431466] LNet: Added LNI 192.168.201.137@tcp43 [8/256/0/180] [ 7644.672680] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [ 7644.684423] LNet: Added LNI 192.168.201.137@tcp44 [8/256/0/180] [ 7647.014742] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [ 7647.029054] LNet: Added LNI 192.168.201.137@tcp45 [8/256/0/180] [ 7649.374862] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [ 7649.385396] LNet: Added LNI 192.168.201.137@tcp46 [8/256/0/180] [ 7651.498855] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [ 7651.508949] LNet: Added LNI 192.168.201.137@tcp47 [8/256/0/180] [ 7653.810497] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [ 7653.819271] LNet: Added LNI 192.168.201.137@tcp48 [8/256/0/180] [ 7656.207275] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [ 7656.220330] LNet: Added LNI 192.168.201.137@tcp49 [8/256/0/180] [ 7658.526026] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [ 7658.536147] LNet: Added LNI 192.168.201.137@tcp50 [8/256/0/180] [ 7667.170018] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7667.171387] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7667.174124] Lustre: Skipped 2 previous similar messages [ 7667.175827] Lustre: Skipped 1 previous similar message [ 7671.644845] Lustre: server umount lustre-MDT0000 complete [ 7672.902627] LustreError: 443324:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773240367 with bad export cookie 17796410976983710542 [ 7672.905178] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7672.908313] LustreError: 443324:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7673.020682] Lustre: server umount lustre-MDT0001 complete [ 7680.062606] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 7682.908742] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7693.791133] Lustre: 442608:0:(client.c:2516:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773240372/real 1773240372] req@000000004b533c47 x1859377171608576/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1773240388 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7693.797183] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7697.887285] LustreError: 442604:0:(client.c:1428:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000059cd00b4 x1859377171609344/t0(0) o250->MGC192.168.201.137@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7697.980723] LustreError: 445065:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7697.985955] LustreError: 445065:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 4 previous similar messages [ 7697.995287] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7698.911068] Lustre: 442606:0:(client.c:2516:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773240377/real 1773240377] req@000000009df4c931 x1859377171608832/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1773240393 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7699.039663] LustreError: 445066:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7699.148782] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 7701.784260] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7703.029031] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 7703.967078] Lustre: 442606:0:(client.c:2516:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773240382/real 1773240382] req@00000000b05658fd x1859377171608960/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1773240398 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7703.972535] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7704.576707] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7705.686417] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7709.151134] Lustre: 442607:0:(client.c:2516:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773240387/real 1773240387] req@00000000ee4abe52 x1859377171609088/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1773240403 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7713.248214] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7713.251144] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7713.254371] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7713.256084] Lustre: Skipped 1 previous similar message [ 7714.272229] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7718.482168] Lustre: server umount lustre-OST0000 complete [ 7722.463809] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7722.466061] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7722.469654] Lustre: Skipped 1 previous similar message [ 7722.471555] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7722.473687] Lustre: Skipped 1 previous similar message [ 7725.851305] Lustre: server umount lustre-MDT0000 complete [ 7727.061260] LustreError: 443323:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773240421 with bad export cookie 17796410976983712537 [ 7727.065146] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7727.065398] LustreError: 443323:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7727.177462] Lustre: server umount lustre-MDT0001 complete [ 7730.617865] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 7731.644645] Key type lgssc unregistered [ 7731.802245] LNet: 470990:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7731.805431] LNetError: 470990:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7731.808663] LNet: Removed LNI 192.168.201.137@tcp [ 7731.809665] LNet: Removed LNI 192.168.201.137@tcp10 [ 7731.810682] LNet: Removed LNI 192.168.201.137@tcp11 [ 7731.811855] LNet: Removed LNI 192.168.201.137@tcp12 [ 7731.813350] LNet: Removed LNI 192.168.201.137@tcp13 [ 7731.815377] LNet: Removed LNI 192.168.201.137@tcp14 [ 7731.816723] LNet: Removed LNI 192.168.201.137@tcp15 [ 7731.817738] LNet: Removed LNI 192.168.201.137@tcp16 [ 7731.818715] LNet: Removed LNI 192.168.201.137@tcp17 [ 7731.820158] LNet: Removed LNI 192.168.201.137@tcp18 [ 7731.821325] LNet: Removed LNI 192.168.201.137@tcp19 [ 7731.823201] LNet: Removed LNI 192.168.201.137@tcp20 [ 7731.824560] LNet: Removed LNI 192.168.201.137@tcp21 [ 7731.825675] LNet: Removed LNI 192.168.201.137@tcp22 [ 7731.827137] LNet: Removed LNI 192.168.201.137@tcp23 [ 7731.829122] LNet: Removed LNI 192.168.201.137@tcp24 [ 7731.831279] LNet: Removed LNI 192.168.201.137@tcp25 [ 7731.832764] LNet: Removed LNI 192.168.201.137@tcp26 [ 7731.833814] LNet: Removed LNI 192.168.201.137@tcp27 [ 7731.835544] LNet: Removed LNI 192.168.201.137@tcp28 [ 7731.836833] LNet: Removed LNI 192.168.201.137@tcp29 [ 7731.838892] LNet: Removed LNI 192.168.201.137@tcp30 [ 7731.840149] LNet: Removed LNI 192.168.201.137@tcp31 [ 7731.841219] LNet: Removed LNI 192.168.201.137@tcp32 [ 7731.842663] LNet: Removed LNI 192.168.201.137@tcp33 [ 7731.843769] LNet: Removed LNI 192.168.201.137@tcp34 [ 7731.845780] LNet: Removed LNI 192.168.201.137@tcp35 [ 7731.846841] LNet: Removed LNI 192.168.201.137@tcp36 [ 7731.848400] LNet: Removed LNI 192.168.201.137@tcp37 [ 7731.849963] LNet: Removed LNI 192.168.201.137@tcp38 [ 7731.851397] LNet: Removed LNI 192.168.201.137@tcp39 [ 7731.853473] LNet: Removed LNI 192.168.201.137@tcp40 [ 7731.854884] LNet: Removed LNI 192.168.201.137@tcp41 [ 7731.856188] LNet: Removed LNI 192.168.201.137@tcp42 [ 7731.857748] LNet: Removed LNI 192.168.201.137@tcp43 [ 7731.858945] LNet: Removed LNI 192.168.201.137@tcp44 [ 7731.860129] LNet: Removed LNI 192.168.201.137@tcp45 [ 7731.861531] LNet: Removed LNI 192.168.201.137@tcp46 [ 7731.862505] LNet: Removed LNI 192.168.201.137@tcp47 [ 7731.863494] LNet: Removed LNI 192.168.201.137@tcp48 [ 7731.864518] LNet: Removed LNI 192.168.201.137@tcp49 [ 7731.871322] LNet: Removed LNI 192.168.201.137@tcp50 [ 7732.136372] Key type .llcrypt unregistered [ 7732.137590] Key type ._llcrypt unregistered [ 7739.349199] Key type ._llcrypt registered [ 7739.350209] Key type .llcrypt registered [ 7739.385414] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 7739.802576] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7739.813975] alg: No test for adler32 (adler32-zlib) [ 7740.666225] Lustre: Lustre: Build Version: 2.17.50_236_gaab9274 [ 7740.741433] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 7742.319216] Key type lgssc registered [ 7742.683619] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7745.199969] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7745.205887] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7746.363644] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7747.605854] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 7750.352753] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7750.485187] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7751.753778] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 7753.453143] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7754.574496] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7756.684343] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7756.795470] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7757.875343] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 7758.586909] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 7760.257392] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7764.788919] loop0: detected capacity change from 0 to 204800000 [ 7764.904687] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7764.908926] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7764.914518] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7766.899966] loop0: detected capacity change from 0 to 204800000 [ 7766.960276] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7766.963452] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7766.966487] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7768.731443] loop0: detected capacity change from 0 to 204800000 [ 7770.614743] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7770.673927] Lustre: fs15246-OST0000: new disk, initializing [ 7770.675869] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [ 7770.697979] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7772.545133] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 7776.545681] loop1: detected capacity change from 0 to 204800000 [ 7778.432106] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7778.467642] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [ 7778.489680] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [ 7778.533900] Lustre: fs15246-MDT0000: new disk, initializing [ 7778.584688] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7778.597906] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7778.645889] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 7779.680691] Lustre: 476571:0:(client.c:1675:after_reply()) @@@ resending request on EINPROGRESS req@000000004bf51aa1 x1859377362398336/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1773240490 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 7779.820434] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 7788.767357] LustreError: 476571:0:(fid_request.c:213:seq_client_alloc_seq()) cli-cli-fs15246-OST0000-osc-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 7788.771809] LustreError: 476571:0:(fid_request.c:255:seq_client_get_seq()) cli-cli-fs15246-OST0000-osc-MDT0000: Can't allocate new sequence: rc = -5 [ 7788.775796] LustreError: 476571:0:(osp_precreate.c:459:osp_precreate_rollover_new_seq()) fs15246-OST0000-osc-MDT0000: alloc fid error: rc = -5 [ 7790.568402] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7790.576631] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [ 7793.965474] Lustre: server umount fs15246-MDT0000 complete [ 7795.685618] LustreError: 473625:0:(ldlm_lib.c:1178:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7795.692618] LustreError: 473625:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 7798.161558] Lustre: server umount fs15246-OST0000 complete [ 7803.871711] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7803.873691] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7803.876993] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7805.921812] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7805.926187] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7811.042259] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7811.046770] Lustre: Skipped 2 previous similar messages [ 7816.159150] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7816.163101] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7816.166276] Lustre: Skipped 1 previous similar message [ 7816.229018] Lustre: server umount lustre-OST0000 complete [ 7821.283331] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7821.290065] Lustre: Skipped 1 previous similar message [ 7824.034023] Lustre: server umount lustre-MDT0000 complete [ 7825.260252] LustreError: 476565:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773240520 with bad export cookie 12110756557419086650 [ 7825.263166] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7825.263770] LustreError: 476565:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7825.380333] Lustre: server umount lustre-MDT0001 complete [ 7829.245874] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 7830.329958] Key type lgssc unregistered [ 7830.457403] LNet: 479631:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7830.460280] LNetError: 479631:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7830.466497] LNet: Removed LNI 192.168.201.137@tcp [ 7830.854469] Key type .llcrypt unregistered [ 7830.856754] Key type ._llcrypt unregistered [ 7837.678574] Key type ._llcrypt registered [ 7837.680311] Key type .llcrypt registered [ 7837.718560] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 7838.022686] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7838.035648] alg: No test for adler32 (adler32-zlib) [ 7838.884520] Lustre: Lustre: Build Version: 2.17.50_236_gaab9274 [ 7838.963974] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 7840.543140] Key type lgssc registered [ 7840.906787] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7854.677571] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_hostid [ 7858.435337] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 7872.222375] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7872.229155] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7873.318406] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7873.329497] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7873.372274] Lustre: lustre-MDT0000: new disk, initializing [ 7873.400877] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7873.409230] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7874.705671] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 7879.652497] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7879.721715] Lustre: 484014:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7879.769847] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7879.773997] Lustre: Skipped 1 previous similar message [ 7879.823551] Lustre: lustre-MDT0001: new disk, initializing [ 7879.881950] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7879.902523] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7879.908800] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7881.508294] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 7884.722991] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7885.880992] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7888.368475] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7888.475548] Lustre: lustre-OST0000: new disk, initializing [ 7888.478670] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7888.513561] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7889.801541] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7889.804628] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7889.836581] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7890.494224] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 7893.497517] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7894.796979] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7894.870754] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7896.143903] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7896.217486] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7898.960431] Lustre: server umount lustre-OST0000 complete [ 7900.639813] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7900.642205] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7900.645938] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7905.248514] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7905.248967] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7909.370428] Lustre: server umount lustre-MDT0000 complete [ 7910.589322] LustreError: 484009:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773240605 with bad export cookie 15061556245643377332 [ 7910.593546] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7910.594784] LustreError: 484009:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7921.206874] Lustre: server umount lustre-MDT0001 complete [ 7929.866814] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_hostid [ 7933.019541] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 7945.177460] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7800 sec ========== 10:50:39 (1773240639) [ 7945.708570] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 10:50:39 (1773240639) === [ 7946.775066] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 10:50:40 (1773240640) === [ 7958.979550] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 7959.961326] Key type lgssc unregistered [ 7960.089365] LNet: 492190:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7960.091433] LNetError: 492190:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7960.102326] LNet: Removed LNI 192.168.201.137@tcp [ 7960.388163] Key type .llcrypt unregistered [ 7960.389366] Key type ._llcrypt unregistered