[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 477777205 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2439 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22D5 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 002295 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2349 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23D9 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2411 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22d5-0xbffe2348] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22d4] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2349-0xbffe23d8] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23d9-0xbffe2410] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2411-0xbffe2438] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.002306] x2apic enabled [ 0.003007] Switched APIC routing to physical x2apic. [ 0.004010] kvm-guest: setup PV IPIs [ 0.007463] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008027] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009013] pid_max: default: 32768 minimum: 301 [ 0.010137] LSM: Security Framework initializing [ 0.011057] Yama: becoming mindful. [ 0.012038] SELinux: Initializing. [ 0.013072] *** VALIDATE selinux *** [ 0.021332] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025790] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027108] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028124] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030009] *** VALIDATE tmpfs *** [ 0.031498] *** VALIDATE proc *** [ 0.032247] *** VALIDATE cgroup *** [ 0.033010] *** VALIDATE cgroup2 *** [ 0.034000] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.034155] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.035010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.036030] Spectre V2 : User space: Vulnerable [ 0.037008] Speculative Store Bypass: Vulnerable [ 0.039910] debug: unmapping init [mem 0xffffffff93859000-0xffffffff93860fff] [ 0.041165] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.042643] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.043024] ... version: 2 [ 0.044016] ... bit width: 48 [ 0.045009] ... generic registers: 4 [ 0.046010] ... value mask: 0000ffffffffffff [ 0.047015] ... max period: 00007fffffffffff [ 0.048008] ... fixed-purpose events: 3 [ 0.048879] ... event mask: 000000070000000f [ 0.049337] rcu: Hierarchical SRCU implementation. [ 0.051467] smp: Bringing up secondary CPUs ... [ 0.052626] x86: Booting SMP configuration: [ 0.053028] .... node #0, CPUs: #1 #2 #3 [ 0.056334] smp: Brought up 1 node, 4 CPUs [ 0.057923] smpboot: Max logical packages: 1 [ 0.058010] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.129000] node 0 deferred pages initialised in 68ms [ 0.133110] devtmpfs: initialized [ 0.134199] x86/mm: Memory block size: 128MB [ 0.136875] gcov: version magic: 0x41383552 [ 0.139282] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.142074] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.144272] pinctrl core: initialized pinctrl subsystem [ 0.145173] [ 0.145634] ************************************************************* [ 0.147010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.148011] ** ** [ 0.150008] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.151009] ** ** [ 0.153013] ** This means that this kernel is built to expose internal ** [ 0.155013] ** IOMMU data structures, which may compromise security on ** [ 0.157014] ** your system. ** [ 0.159009] ** ** [ 0.162010] ** If you see this message and you are not debugging the ** [ 0.164011] ** kernel, report this immediately to your vendor! ** [ 0.166010] ** ** [ 0.167010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.169014] ************************************************************* [ 0.171633] NET: Registered protocol family 16 [ 0.173417] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.176070] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.178084] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.182119] cpuidle: using governor menu [ 0.183520] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.185394] PCI: Using configuration type 1 for base access [ 0.186170] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.192130] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.195066] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.199065] cryptd: max_cpu_qlen set to 1000 [ 0.202254] ACPI: Added _OSI(Module Device) [ 0.203011] ACPI: Added _OSI(Processor Device) [ 0.203929] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.206035] ACPI: Added _OSI(Processor Aggregator Device) [ 0.211869] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.218143] ACPI: Interpreter enabled [ 0.220082] ACPI: PM: (supports S0 S3 S4 S5) [ 0.221012] ACPI: Using IOAPIC for interrupt routing [ 0.223097] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.226426] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.238742] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.240044] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.243021] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.246088] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.251586] acpiphp: Slot [2] registered [ 0.253132] acpiphp: Slot [5] registered [ 0.255114] acpiphp: Slot [6] registered [ 0.256130] acpiphp: Slot [7] registered [ 0.258316] acpiphp: Slot [8] registered [ 0.260339] acpiphp: Slot [9] registered [ 0.262164] acpiphp: Slot [10] registered [ 0.265178] acpiphp: Slot [3] registered [ 0.267186] acpiphp: Slot [4] registered [ 0.269108] acpiphp: Slot [11] registered [ 0.270137] acpiphp: Slot [12] registered [ 0.272135] acpiphp: Slot [13] registered [ 0.273118] acpiphp: Slot [14] registered [ 0.275137] acpiphp: Slot [15] registered [ 0.276204] acpiphp: Slot [16] registered [ 0.278120] acpiphp: Slot [17] registered [ 0.279203] acpiphp: Slot [18] registered [ 0.280000] acpiphp: Slot [19] registered [ 0.280000] acpiphp: Slot [20] registered [ 0.282105] acpiphp: Slot [21] registered [ 0.283146] acpiphp: Slot [22] registered [ 0.285087] acpiphp: Slot [23] registered [ 0.286082] acpiphp: Slot [24] registered [ 0.287089] acpiphp: Slot [25] registered [ 0.288137] acpiphp: Slot [26] registered [ 0.289000] acpiphp: Slot [27] registered [ 0.289000] acpiphp: Slot [28] registered [ 0.290119] acpiphp: Slot [29] registered [ 0.292117] acpiphp: Slot [30] registered [ 0.293141] acpiphp: Slot [31] registered [ 0.294065] PCI host bridge to bus 0000:00 [ 0.296024] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.298034] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.300043] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.303025] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.306026] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.309036] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.311225] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.313917] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.317288] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.328009] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.333577] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.336017] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.339018] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.341016] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.345378] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.347753] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.350043] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.353884] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.360017] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.374020] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.378014] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.381000] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.391016] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.398021] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.412020] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.422901] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.428016] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.440040] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.459022] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.470573] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.480021] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.487018] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.506030] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.519272] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.525036] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.531018] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.546024] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.556408] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.564017] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.570015] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.583019] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.592972] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.599017] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.604014] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.619017] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.629458] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.633396] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.635386] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.637359] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.640256] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.643342] iommu: Default domain type: Passthrough [ 0.645720] SCSI subsystem initialized [ 0.647179] ACPI: bus type USB registered [ 0.649146] usbcore: registered new interface driver usbfs [ 0.652114] usbcore: registered new interface driver hub [ 0.654105] usbcore: registered new device driver usb [ 0.656269] pps_core: LinuxPPS API ver. 1 registered [ 0.658012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.661070] PTP clock support registered [ 0.663045] EDAC MC: Ver: 3.0.0 [ 0.664134] PCI: Using ACPI for IRQ routing [ 0.665796] NetLabel: Initializing [ 0.667013] NetLabel: domain hash size = 128 [ 0.669013] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.671082] NetLabel: unlabeled traffic allowed by default [ 0.672305] vgaarb: loaded [ 0.674369] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.677017] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.681459] clocksource: Switched to clocksource kvm-clock [ 0.793388] VFS: Disk quotas dquot_6.6.0 [ 0.794734] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.797168] *** VALIDATE ramfs *** [ 0.798322] *** VALIDATE hugetlbfs *** [ 0.799828] pnp: PnP ACPI init [ 0.802159] pnp: PnP ACPI: found 6 devices [ 0.817295] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.820740] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.823036] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.825355] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.827414] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.829914] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.832680] NET: Registered protocol family 2 [ 0.835208] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.840872] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.844864] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.850739] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.853364] TCP: Hash tables configured (established 65536 bind 65536) [ 0.855584] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.857818] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.859951] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.862824] NET: Registered protocol family 1 [ 0.865172] RPC: Registered named UNIX socket transport module. [ 0.867413] RPC: Registered udp transport module. [ 0.869149] RPC: Registered tcp transport module. [ 0.870731] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.873143] NET: Registered protocol family 44 [ 0.874730] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.877263] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.879595] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.882230] PCI: CLS 0 bytes, default 64 [ 0.884683] Unpacking initramfs... [ 2.252266] debug: unmapping init [mem 0xffff8ec9fcc54000-0xffff8ec9fffbffff] [ 2.255284] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.257020] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.259120] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.773260] Initialise system trusted keyrings [ 2.775115] Key type blacklist registered [ 2.777085] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.785839] zbud: loaded [ 2.789099] *** VALIDATE nfs *** [ 2.790424] *** VALIDATE nfs4 *** [ 2.792972] pstore: using deflate compression [ 2.797133] Platform Keyring initialized [ 2.899054] NET: Registered protocol family 38 [ 2.901296] Key type asymmetric registered [ 2.903196] Asymmetric key parser 'x509' registered [ 2.905759] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.908747] io scheduler mq-deadline registered [ 2.910315] io scheduler kyber registered [ 2.911881] io scheduler bfq registered [ 2.913852] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.916784] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.919708] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.922756] ACPI: Power Button [PWRF] [ 2.928120] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.934798] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.950596] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.958177] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.974803] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.002641] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.030272] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.039242] Non-volatile memory driver v1.3 [ 3.041332] Linux agpgart interface v0.103 [ 3.074630] virtio_blk virtio1: [vda] 133936 512-byte logical blocks (68.6 MB/65.4 MiB) [ 3.077690] vda: detected capacity change from 0 to 68575232 [ 3.092220] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.095244] vdb: detected capacity change from 0 to 1073741824 [ 3.111110] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.114115] vdc: detected capacity change from 0 to 2621440000 [ 3.129153] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.132093] vdd: detected capacity change from 0 to 2621440000 [ 3.151429] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.154557] vde: detected capacity change from 0 to 4294967296 [ 3.177032] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.180369] vdf: detected capacity change from 0 to 4294967296 [ 3.189993] libphy: Fixed MDIO Bus: probed [ 3.196460] usbcore: registered new interface driver usbserial_generic [ 3.199275] usbserial: USB Serial support registered for generic [ 3.201905] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.206029] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.207776] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.210203] mousedev: PS/2 mouse device common for all mice [ 3.213293] rtc_cmos 00:05: RTC can wake from S4 [ 3.216728] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.216860] rtc_cmos 00:05: registered as rtc0 [ 3.221953] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.225229] intel_pstate: CPU model not supported [ 3.228560] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.231303] hid: raw HID events driver (C) Jiri Kosina [ 3.234344] usbcore: registered new interface driver usbhid [ 3.234483] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.236479] usbhid: USB HID core driver [ 3.236641] drop_monitor: Initializing network drop monitor service [ 3.244110] Initializing XFRM netlink socket [ 3.246238] NET: Registered protocol family 10 [ 3.249536] Segment Routing with IPv6 [ 3.251176] NET: Registered protocol family 17 [ 3.253396] mpls_gso: MPLS GSO support [ 3.259410] RAS: Correctable Errors collector initialized. [ 3.261573] AVX version of gcm_enc/dec engaged. [ 3.262995] AES CTR mode by8 optimization enabled [ 3.338157] sched_clock: Marking stable (3338128732, 0)->(4131658303, -793529571) [ 3.341427] registered taskstats version 1 [ 3.343817] Loading compiled-in X.509 certificates [ 3.345695] zswap: loaded using pool lzo/zbud [ 3.368333] Key type big_key registered [ 3.381215] Key type encrypted registered [ 3.382931] ima: No TPM chip found, activating TPM-bypass! [ 3.384800] ima: Allocated hash algorithm: sha1 [ 3.385936] ima: No architecture policies found [ 3.387843] evm: Initialising EVM extended attributes: [ 3.390189] evm: security.selinux [ 3.391480] evm: security.ima [ 3.392650] evm: security.capability [ 3.394163] evm: HMAC attrs: 0x1 [ 3.396616] rtc_cmos 00:05: setting system clock to 2026-04-03 11:30:44 UTC (1775215844) [ 3.403167] debug: unmapping init [mem 0xffffffff94803000-0xffffffff949fffff] [ 3.406638] debug: unmapping init [mem 0xffffffff93582000-0xffffffff93858fff] [ 3.415078] Write protecting the kernel read-only data: 28672k [ 3.418672] debug: unmapping init [mem 0xffffffff91c03000-0xffffffff91dfffff] [ 3.421274] debug: unmapping init [mem 0xffffffff92514000-0xffffffff925fffff] [ 3.459790] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.467592] systemd[1]: Detected virtualization kvm. [ 3.469460] systemd[1]: Detected architecture x86-64. [ 3.471566] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.499896] systemd[1]: No hostname configured. [ 3.501326] systemd[1]: Set hostname to . [ 3.503583] random: systemd: uninitialized urandom read (16 bytes read) [ 3.506145] systemd[1]: Initializing machine ID from random generator. [ 3.553909] random: ln: uninitialized urandom read (6 bytes read) [ 3.626305] random: systemd: uninitialized urandom read (16 bytes read) [ 3.629732] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.633810] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.637348] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Starting Setup Virtual Console... [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Initrd Root Device. Starting Apply Kernel Variables... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Swap. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.204106] device-mapper: uevent: version 1.0.3 [ 4.206057] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.847185] virtio_net virtio0 ens2: renamed from eth0 [ 4.861456] random: fast init done [ 4.912079] scsi host0: ata_piix [ 4.926426] scsi host1: ata_piix [ 4.927662] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 4.929341] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.567249] dracut-initqueue[572]: RTNETLINK answers: File exists [ 9.901242] random: crng init done [ 9.902531] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.165236] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.224640] printk: systemd: 25 output lines suppressed due to ratelimiting [ 11.448838] SELinux: Disabled at runtime. [ 11.504941] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.512553] systemd[1]: Detected virtualization kvm. [ 11.513984] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 11.985365] systemd[1]: initrd-switch-root.service: Succeeded. [ 11.988919] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 11.994758] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 11.999259] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.004273] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.013594] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.020070] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Mounting Kernel Debug File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting POSIX Message Queue File System... Starting Remount Root and Kernel File Systems... [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice User and Session Slice. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on udev Control Socket. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Reached target Slices. [ OK ] Created slice system-getty.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... Starting Create list of required st…ce nodes for the current kernel... Mounting Huge Pages File System... [ OK ] Created slice system-serial\x2dgetty.slice. Starting Apply Kernel Variables... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Activating swap /dev/disk/by-label/SWAP... [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ 12.225364] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 12.445441] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.693714] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.700892] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.846185] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 12.855503] EDAC sbridge: Ver: 1.1.2 [ 16.153879] Key type dns_resolver registered [ 16.999313] NFS: Registering the id_resolver key type [ 17.006180] Key type id_resolver registered [ 17.011722] Key type id_legacy registered [* ] A start job is running for Configur…-only root support (5s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg241-server login: [ 76.076380] hrtimer: interrupt took 6428269 ns [ 88.972311] libcfs: loading out-of-tree module taints kernel. [ 89.055289] Key type ._llcrypt registered [ 89.059103] Key type .llcrypt registered [ 89.248987] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_hostid [ 108.581434] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 110.997442] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 111.008800] alg: No test for adler32 (adler32-zlib) [ 112.633966] Lustre: Lustre: Build Version: 2.17.51_74_g8e7ba6b [ 113.315268] LNet: Added LNI 192.168.202.141@tcp [8/256/0/180] [ 115.055923] Key type lgssc registered [ 116.108405] Lustre: Echo OBD driver; http://www.lustre.org/ [ 134.755150] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 182.625586] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 199.268704] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 199.293297] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 200.580804] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 200.626307] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 200.727390] Lustre: lustre-MDT0000: new disk, initializing [ 200.807129] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 200.848575] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 205.774354] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 220.216395] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 220.329241] Lustre: 6497:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 220.358108] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 220.362440] Lustre: Skipped 1 previous similar message [ 220.452849] Lustre: lustre-MDT0001: new disk, initializing [ 220.554369] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 220.619513] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 220.643300] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 225.620145] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 230.530183] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 240.750593] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 240.958863] Lustre: lustre-OST0000: new disk, initializing [ 240.962862] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 241.028107] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 243.237404] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 243.251713] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 243.290146] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 247.982506] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 264.917447] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 265.162888] Lustre: lustre-OST0001: new disk, initializing [ 265.171217] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 265.304426] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 270.931261] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 270.952526] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 271.024626] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 271.875542] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 285.305797] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 293.933499] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 306.465474] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing check_logdir /tmp/testlogs/ [ 312.667130] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing yml_node [ 317.571499] Lustre: DEBUG MARKER: Client: 2.17.51.74 [ 320.021655] Lustre: DEBUG MARKER: MDS: 2.17.51.74 [ 323.280234] Lustre: DEBUG MARKER: OSS: 2.17.51.74 [ 324.965714] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Apr 3 07:36:04 EDT 2026 [ 344.426681] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 346.025371] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 363.488228] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 363.491342] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 363.507177] Lustre: Skipped 1 previous similar message [ 363.523781] Lustre: Skipped 3 previous similar messages [ 367.651721] Lustre: server umount lustre-MDT0000 complete [ 371.911343] LustreError: 10259:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775216213 with bad export cookie 3265392108273084561 [ 371.926518] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 371.928624] LustreError: 10259:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 373.736207] LustreError: 6505:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 373.738914] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 373.753795] LustreError: 6505:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 373.754453] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 373.791677] Lustre: Skipped 3 previous similar messages [ 376.800267] LustreError: 6504:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 376.802058] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 376.817057] LustreError: 6504:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 376.854199] Lustre: Skipped 2 previous similar messages [ 378.454305] Lustre: server umount lustre-MDT0001 complete [ 382.380374] Lustre: server umount lustre-OST0000 complete [ 386.732899] Lustre: server umount lustre-OST0001 complete [ 394.335201] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_hostid [ 403.679754] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 440.929288] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 441.199224] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 441.231830] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 441.419883] Lustre: lustre-MDT0000: new disk, initializing [ 441.506481] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 441.546255] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 446.469637] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 457.266838] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 457.392652] Lustre: 14980:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 457.484584] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 457.493199] Lustre: Skipped 1 previous similar message [ 457.688713] Lustre: lustre-MDT0001: new disk, initializing [ 457.913499] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 457.993577] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 458.013094] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 463.338656] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 471.599466] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 476.038991] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 483.421522] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 483.627440] Lustre: lustre-OST0000: new disk, initializing [ 483.630384] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 483.691173] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 484.826162] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 484.841367] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 484.982285] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 489.312058] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 497.410824] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 502.058649] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 502.288704] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 508.011668] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 508.300735] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 510.435830] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 510.437808] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 510.443726] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 510.446439] LustreError: Skipped 1 previous similar message [ 510.478338] Lustre: Skipped 1 previous similar message [ 515.557109] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 515.559403] Lustre: Skipped 1 previous similar message [ 516.583750] Lustre: server umount lustre-OST0000 complete [ 524.257622] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 524.268788] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 524.307376] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 524.328041] Lustre: Skipped 1 previous similar message [ 527.480841] Lustre: server umount lustre-MDT0000 complete [ 530.914211] LustreError: 14993:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 530.925661] LustreError: 14993:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 531.590141] LustreError: 15769:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775216372 with bad export cookie 3265392108273086094 [ 531.598076] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 531.606588] LustreError: 15769:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 531.998462] Lustre: server umount lustre-MDT0001 complete [ 536.845031] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 07:39:36 (1775216376) [ 539.962408] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 541.836675] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 07:39:41 (1775216381) [ 548.480872] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 548.865184] LustreError: 18985:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 548.926480] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 553.951473] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 553.956440] LustreError: 18986:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 563.173144] LustreError: 18985:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 563.204724] LustreError: 18985:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 563.771885] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 568.604381] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 574.084929] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 578.763736] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 586.278780] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 595.275803] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 603.159947] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 614.368615] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 614.375076] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 614.389213] Lustre: Skipped 2 previous similar messages [ 614.401336] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 614.410278] Lustre: Skipped 3 previous similar messages [ 619.373612] Lustre: server umount lustre-OST0000 complete [ 624.609383] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 629.872947] Lustre: server umount lustre-MDT0000 complete [ 634.034181] LustreError: 18967:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775216475 with bad export cookie 3265392108273086976 [ 634.044743] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 634.056952] LustreError: 18967:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 634.499320] Lustre: server umount lustre-MDT0001 complete [ 647.259859] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing unload_modules_local [ 650.499199] Key type lgssc unregistered [ 650.909469] LNet: 22166:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 650.934536] LNetError: 22166:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 652.013040] LNet: Removed LNI 192.168.202.141@tcp [ 653.254325] Key type .llcrypt unregistered [ 653.258899] Key type ._llcrypt unregistered [ 679.422522] Key type ._llcrypt registered [ 679.425588] Key type .llcrypt registered [ 679.586563] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 680.693196] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 680.855511] alg: No test for adler32 (adler32-zlib) [ 682.038976] Lustre: Lustre: Build Version: 2.17.51_74_g8e7ba6b [ 682.268024] LNet: Added LNI 192.168.202.141@tcp [8/256/0/180] [ 683.967395] Key type lgssc registered [ 684.953917] Lustre: Echo OBD driver; http://www.lustre.org/ [ 693.914358] loop: module loaded [ 694.081101] loop0: detected capacity change from 0 to 204800000 [ 694.374386] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 694.388877] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 694.586614] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 700.803553] loop0: detected capacity change from 0 to 204800000 [ 700.982393] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 700.996788] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 701.010205] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 707.230083] loop0: detected capacity change from 0 to 204800000 [ 713.860735] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 713.897847] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 715.394555] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 715.486441] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 715.565978] Lustre: lustre-MDT0000: new disk, initializing [ 715.668339] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 715.685757] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 720.164208] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 730.165918] loop1: detected capacity change from 0 to 204800000 [ 736.041142] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 736.437473] Lustre: lustre-OST07c6: new disk, initializing [ 736.447078] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 736.458735] Lustre: Skipped 1 previous similar message [ 736.543050] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 743.365550] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 746.156406] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 746.167796] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 746.215588] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 749.402080] Lustre: 24327:0:(osd_internal.h:1446:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 749.408196] Lustre: 24327:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 749.412566] Lustre: 24327:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 749.416909] Lustre: 24327:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 749.423560] Lustre: 24327:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 749.428193] Lustre: 24327:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 766.894379] Lustre: 24328:0:(osd_internal.h:1446:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 766.900380] Lustre: 24328:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 766.904518] Lustre: 24328:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 766.910723] Lustre: 24328:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/4 [ 766.915605] Lustre: 24328:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 766.920579] Lustre: 24328:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 771.681701] Lustre: 25588:0:(osd_internal.h:1446:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 771.696599] Lustre: 25588:0:(osd_internal.h:1446:osd_trans_exec_op()) Skipped 1 previous similar message [ 771.706762] Lustre: 25588:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 771.716788] Lustre: 25588:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 771.727553] Lustre: 25588:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 771.733949] Lustre: 25588:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 771.742710] Lustre: 25588:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/23/0, punch: 0/0/0, quota 4/150/0 [ 771.749994] Lustre: 25588:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 771.760611] Lustre: 25588:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 771.773954] Lustre: 25588:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 771.783703] Lustre: 25588:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 771.790679] Lustre: 25588:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 775.003758] Lustre: 24328:0:(osd_internal.h:1446:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 775.013869] Lustre: 24328:0:(osd_internal.h:1446:osd_trans_exec_op()) Skipped 1 previous similar message [ 775.032258] Lustre: 24328:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 775.050718] Lustre: 24328:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 775.061278] Lustre: 24328:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 775.075738] Lustre: 24328:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 775.086588] Lustre: 24328:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/0 [ 775.099797] Lustre: 24328:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 775.113885] Lustre: 24328:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 775.127936] Lustre: 24328:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 775.137649] Lustre: 24328:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 775.147730] Lustre: 24328:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 779.696781] Lustre: 24327:0:(osd_internal.h:1446:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 779.717511] Lustre: 24327:0:(osd_internal.h:1446:osd_trans_exec_op()) Skipped 1 previous similar message [ 779.726590] Lustre: 24327:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 779.741355] Lustre: 24327:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 779.751391] Lustre: 24327:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 779.756473] Lustre: 24327:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 779.761632] Lustre: 24327:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/2 [ 779.768760] Lustre: 24327:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 779.776709] Lustre: 24327:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 779.785802] Lustre: 24327:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 779.796606] Lustre: 24327:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 779.808255] Lustre: 24327:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 790.977089] Lustre: 25588:0:(osd_internal.h:1446:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 790.988606] Lustre: 25588:0:(osd_internal.h:1446:osd_trans_exec_op()) Skipped 5 previous similar messages [ 791.003619] Lustre: 25588:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 791.016560] Lustre: 25588:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 791.033654] Lustre: 25588:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 791.044738] Lustre: 25588:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 791.051773] Lustre: 25588:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/2 [ 791.061563] Lustre: 25588:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 791.066881] Lustre: 25588:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 791.071699] Lustre: 25588:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 791.079453] Lustre: 25588:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 791.086953] Lustre: 25588:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 838.118657] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 838.140280] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 843.236261] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 844.066856] Lustre: server umount lustre-OST07c6 complete [ 854.521631] Lustre: server umount lustre-MDT0000 complete [ 871.859158] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing unload_modules_local [ 875.751498] Key type lgssc unregistered [ 876.072323] LNet: 27843:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 876.093251] LNetError: 27843:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 876.127918] LNet: Removed LNI 192.168.202.141@tcp [ 877.256506] Key type .llcrypt unregistered [ 877.259535] Key type ._llcrypt unregistered [ 898.318171] Key type ._llcrypt registered [ 898.319827] Key type .llcrypt registered [ 898.513460] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 899.346526] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 899.548934] alg: No test for adler32 (adler32-zlib) [ 900.985764] Lustre: Lustre: Build Version: 2.17.51_74_g8e7ba6b [ 901.174307] LNet: Added LNI 192.168.202.141@tcp [8/256/0/180] [ 902.895180] Key type lgssc registered [ 904.469262] Lustre: Echo OBD driver; http://www.lustre.org/ [ 914.095314] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 914.114300] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 916.296579] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 921.935824] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 926.689889] LustreError: 29100:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 926.713966] LustreError: 29100:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 931.316809] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 931.820491] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 932.033109] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 938.087244] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 945.783537] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 949.890945] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 958.429947] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 958.802396] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 961.930514] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 965.839864] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 972.520647] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 992.736557] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 992.740232] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 992.759310] LustreError: Skipped 1 previous similar message [ 992.780125] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 992.792390] Lustre: Skipped 1 previous similar message [ 992.819110] Lustre: Skipped 1 previous similar message [ 995.765185] Lustre: server umount lustre-OST0000 complete [ 1002.979434] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1002.984141] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1002.993444] Lustre: Skipped 1 previous similar message [ 1003.011286] Lustre: Skipped 1 previous similar message [ 1006.625286] Lustre: server umount lustre-MDT0000 complete [ 1008.096895] LustreError: 29100:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1008.114411] LustreError: 29100:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1010.695995] LustreError: 29082:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775216851 with bad export cookie 7138114920114567735 [ 1010.709557] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1010.997878] Lustre: server umount lustre-MDT0001 complete [ 1022.816748] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing unload_modules_local [ 1025.461125] Key type lgssc unregistered [ 1025.781822] LNet: 32486:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1025.808821] LNetError: 32486:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1025.826755] LNet: Removed LNI 192.168.202.141@tcp [ 1026.722573] Key type .llcrypt unregistered [ 1026.728084] Key type ._llcrypt unregistered [ 1049.046522] Key type ._llcrypt registered [ 1049.049044] Key type .llcrypt registered [ 1049.185685] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 1050.549825] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1050.681408] alg: No test for adler32 (adler32-zlib) [ 1051.903923] Lustre: Lustre: Build Version: 2.17.51_74_g8e7ba6b [ 1052.253271] LNet: Added LNI 192.168.202.141@tcp [8/256/0/180] [ 1053.951181] Key type lgssc registered [ 1055.104991] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1063.657578] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1063.681502] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1065.134757] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1070.703524] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 1075.700692] LustreError: 33739:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1075.740427] LustreError: 33739:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1080.803744] LustreError: 33740:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1081.477301] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1082.067507] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1086.762909] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 1093.641347] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1098.590763] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1106.282678] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1106.598305] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1110.739778] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1113.747961] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 1119.617729] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1131.489881] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1131.495218] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1131.516693] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1131.549236] Lustre: Skipped 1 previous similar message [ 1136.079824] Lustre: server umount lustre-OST0000 complete [ 1141.731804] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1141.743681] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1141.751079] Lustre: Skipped 2 previous similar messages [ 1146.852306] LustreError: 33740:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1146.864926] LustreError: 33740:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1146.869122] Lustre: server umount lustre-MDT0000 complete [ 1150.948682] LustreError: 33735:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1150.973765] LustreError: 33735:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1151.050583] LustreError: 33719:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775216992 with bad export cookie 7662714069752058618 [ 1151.063514] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1151.501224] Lustre: server umount lustre-MDT0001 complete [ 1163.049401] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing unload_modules_local [ 1166.135183] Key type lgssc unregistered [ 1166.552681] LNet: 36920:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1166.564956] LNetError: 36920:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1166.601059] LNet: Removed LNI 192.168.202.141@tcp [ 1167.569405] Key type .llcrypt unregistered [ 1167.571351] Key type ._llcrypt unregistered [ 1195.610488] Key type ._llcrypt registered [ 1195.613533] Key type .llcrypt registered [ 1195.742935] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 1197.942477] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1198.009086] alg: No test for adler32 (adler32-zlib) [ 1199.069541] Lustre: Lustre: Build Version: 2.17.51_74_g8e7ba6b [ 1199.522687] LNet: Added LNI 192.168.202.141@tcp [8/256/0/180] [ 1201.295174] Key type lgssc registered [ 1202.586761] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1211.865954] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1211.878755] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1213.405830] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1218.094287] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 1223.651126] LustreError: 38338:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1223.671353] LustreError: 38338:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1226.658262] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1226.999203] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1231.417902] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 1237.615626] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1242.059160] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1250.633576] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1251.161293] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1258.376202] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1258.896802] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 1265.617977] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1272.749992] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.41@tcp (stopping) [ 1273.314463] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1273.342197] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1273.371209] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1277.830125] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.41@tcp (stopping) [ 1277.853398] Lustre: Skipped 2 previous similar messages [ 1278.447825] Lustre: server umount lustre-MDT0000 complete [ 1278.944548] LustreError: 38338:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1278.963593] LustreError: 38338:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1282.452916] LustreError: 38318:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775217123 with bad export cookie 16702592442181913036 [ 1282.461948] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1282.468959] LustreError: 38318:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1282.951711] LustreError: 38332:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.41@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1282.971579] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.41@tcp (stopping) [ 1284.102052] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1284.142375] Lustre: Skipped 2 previous similar messages [ 1287.079939] LustreError: 39498:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1287.080027] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1287.097527] LustreError: 39498:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1287.102533] Lustre: Skipped 1 previous similar message [ 1288.896751] Lustre: server umount lustre-MDT0001 complete [ 1318.367129] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1318.647493] Lustre: server umount lustre-OST0000 complete [ 1336.010334] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing unload_modules_local [ 1338.987471] Key type lgssc unregistered [ 1339.364410] LNet: 41848:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1339.374970] LNetError: 41848:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1339.396475] LNet: Removed LNI 192.168.202.141@tcp [ 1340.255977] Key type .llcrypt unregistered [ 1340.258480] Key type ._llcrypt unregistered [ 1366.740807] Key type ._llcrypt registered [ 1366.744816] Key type .llcrypt registered [ 1366.898455] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 1368.672903] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1368.755784] alg: No test for adler32 (adler32-zlib) [ 1369.792503] Lustre: Lustre: Build Version: 2.17.51_74_g8e7ba6b [ 1370.176089] LNet: Added LNI 192.168.202.141@tcp [8/256/0/180] [ 1371.903253] Key type lgssc registered [ 1373.619233] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1381.928903] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1381.962971] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1383.643329] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1389.395398] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 1393.127992] LustreError: 43253:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1393.142178] LustreError: 43253:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1398.251833] LustreError: 43252:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1399.313401] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1399.987287] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1405.485799] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 1411.353620] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1415.026317] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1422.991534] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1423.343752] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1428.474444] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1430.542957] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 1437.290733] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1443.822054] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1443.843929] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1443.845421] Lustre: Skipped 1 previous similar message [ 1448.928307] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1448.956155] Lustre: Skipped 3 previous similar messages [ 1449.146687] Lustre: server umount lustre-OST0000 complete [ 1467.595532] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1467.618785] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1467.626845] Lustre: Skipped 1 previous similar message [ 1472.999225] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1473.011876] Lustre: Skipped 1 previous similar message [ 1474.017031] LustreError: 44954:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1474.043165] LustreError: 44954:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1474.165665] Lustre: server umount lustre-MDT0000 complete [ 1478.237541] LustreError: 43232:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775217319 with bad export cookie 14557873567272344868 [ 1478.239079] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1478.248016] LustreError: 43232:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1484.969676] Lustre: server umount lustre-MDT0001 complete [ 1500.509392] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing unload_modules_local [ 1505.597065] Key type lgssc unregistered [ 1506.067386] LNet: 46609:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1506.092478] LNetError: 46609:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1506.117042] LNet: Removed LNI 192.168.202.141@tcp [ 1507.159931] Key type .llcrypt unregistered [ 1507.162649] Key type ._llcrypt unregistered [ 1534.035336] Key type ._llcrypt registered [ 1534.037922] Key type .llcrypt registered [ 1534.135954] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 1535.878691] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1535.970408] alg: No test for adler32 (adler32-zlib) [ 1537.245218] Lustre: Lustre: Build Version: 2.17.51_74_g8e7ba6b [ 1537.579564] LNet: Added LNI 192.168.202.141@tcp [8/256/0/180] [ 1539.384205] Key type lgssc registered [ 1540.647099] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1549.019274] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1549.069574] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1550.762353] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1555.569884] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 1560.038773] LustreError: 48009:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1560.061704] LustreError: 48009:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1565.155299] LustreError: 48008:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1565.441979] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1565.822695] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1570.483621] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 1575.735056] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1579.776953] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1587.383555] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1587.788918] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1594.959607] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1595.194474] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 1601.259025] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1607.460287] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1611.633359] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1624.330294] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-04-03 7h57m43s [ 1625.730670] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1628.597168] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.41@tcp (stopping) [ 1629.638862] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.41@tcp (stopping) [ 1630.689293] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1630.704264] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1630.712178] Lustre: Skipped 2 previous similar messages [ 1630.737487] Lustre: Skipped 2 previous similar messages [ 1634.165660] Lustre: server umount lustre-MDT0000 complete [ 1634.698879] LustreError: 48004:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.41@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1637.880606] LustreError: 48008:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1637.912251] LustreError: 48008:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1638.068726] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1644.011123] LustreError: 48009:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1644.026319] LustreError: 48009:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1645.482407] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1645.586468] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1645.919888] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1646.003683] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1646.950560] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1650.400753] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 1653.923822] Lustre: DEBUG MARKER: Wait for df (36119) ... [ 1655.736714] Lustre: DEBUG MARKER: done [ 1661.409452] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1661.421599] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1661.434908] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1661.439466] Lustre: Skipped 3 previous similar messages [ 1662.441542] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1662.449096] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1667.553533] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1667.565788] Lustre: Skipped 2 previous similar messages [ 1675.231300] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1675.413573] Lustre: server umount lustre-OST0000 complete [ 1682.916107] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1682.916973] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1682.929645] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1682.944835] Lustre: Skipped 1 previous similar message [ 1682.973962] Lustre: Skipped 4 previous similar messages [ 1694.175189] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1694.528289] Lustre: server umount lustre-MDT0000 complete [ 1696.250995] LustreError: 48008:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1696.286975] LustreError: 48008:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1698.851699] LustreError: 48725:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775217539 with bad export cookie 3440486488437976195 [ 1698.885118] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1699.383649] Lustre: server umount lustre-MDT0001 complete [ 1712.234532] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing unload_modules_local [ 1715.000793] Key type lgssc unregistered [ 1715.409381] LNet: 52572:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1715.418150] LNetError: 52572:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1716.458724] LNet: Removed LNI 192.168.202.141@tcp [ 1717.753654] Key type .llcrypt unregistered [ 1717.754842] Key type ._llcrypt unregistered [ 1763.431916] Key type ._llcrypt registered [ 1763.433836] Key type .llcrypt registered [ 1763.557288] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 1764.793746] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1764.842841] alg: No test for adler32 (adler32-zlib) [ 1765.839422] Lustre: Lustre: Build Version: 2.17.51_74_g8e7ba6b [ 1765.957362] LNet: Added LNI 192.168.202.141@tcp [8/256/0/180] [ 1767.672105] Key type lgssc registered [ 1769.125740] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1778.988330] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1779.035592] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1780.431444] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1780.447662] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1780.624788] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1786.058451] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 1796.513193] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1796.616192] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1796.628884] Lustre: 54767:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1796.906411] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1803.424854] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 1811.294974] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1817.177491] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1824.852649] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1825.083495] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1825.264573] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1830.470322] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1831.810795] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 1838.264799] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1844.583848] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-04-03 8h01m24s [ 1846.808480] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1856.228900] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1859.776769] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1859.851724] LustreError: 54773:0:(mdt_handler.c:2347:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1879.990818] Lustre: lustre-MDT0000: Client 6ecb742f-33ee-4417-9dc4-297b5c9be1ba (at 192.168.202.41@tcp) reconnecting [ 1899.887129] LustreError: 54773:0:(mdt_handler.c:2347:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1902.984516] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1907.021821] Lustre: DEBUG MARKER: done [ 1917.408930] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1917.424061] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1917.458506] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1921.118698] Lustre: server umount lustre-OST0000 complete [ 1927.649588] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1927.657305] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1927.661606] Lustre: Skipped 2 previous similar messages [ 1927.666350] Lustre: Skipped 1 previous similar message [ 1928.779149] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1928.801860] Lustre: Skipped 2 previous similar messages [ 1934.636320] Lustre: server umount lustre-MDT0000 complete [ 1939.586318] LustreError: 54758:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775217780 with bad export cookie 15560677473747240919 [ 1939.588871] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1939.608484] LustreError: 54758:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1940.088181] Lustre: server umount lustre-MDT0001 complete [ 1954.491248] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing unload_modules_local [ 1958.269508] Key type lgssc unregistered [ 1958.722843] LNet: 58755:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1958.734892] LNetError: 58755:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1958.755698] LNet: Removed LNI 192.168.202.141@tcp [ 1959.719575] Key type .llcrypt unregistered [ 1959.721443] Key type ._llcrypt unregistered [ 2008.930451] Key type ._llcrypt registered [ 2008.932708] Key type .llcrypt registered [ 2009.037446] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 2009.928876] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2010.025234] alg: No test for adler32 (adler32-zlib) [ 2011.262779] Lustre: Lustre: Build Version: 2.17.51_74_g8e7ba6b [ 2011.417973] LNet: Added LNI 192.168.202.141@tcp [8/256/0/180] [ 2013.066490] Key type lgssc registered [ 2014.530509] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2018.366521] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2018.393993] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2019.775149] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2019.825161] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2020.099472] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2021.392476] Lustre: Failing over lustre-MDT0000 [ 2021.693678] Lustre: server umount lustre-MDT0000 complete [ 2029.171411] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 08:04:28 (1775217868) [ 2036.300375] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2036.792961] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2042.181169] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 2051.225566] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2051.375793] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2051.468081] Lustre: 61538:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2051.763817] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2057.090640] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 2065.880540] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2071.262112] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2079.603972] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2079.857603] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2080.077706] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2087.313738] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 2088.693102] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2094.555346] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2100.698898] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2109.413713] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2109.420340] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2109.433198] Lustre: Skipped 2 previous similar messages [ 2109.456340] Lustre: Skipped 2 previous similar messages [ 2114.532281] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2114.541178] Lustre: Skipped 2 previous similar messages [ 2116.950210] Lustre: server umount lustre-MDT0000 complete [ 2121.467417] LustreError: 62258:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775217962 with bad export cookie 770550640034833988 [ 2121.473504] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2121.490368] LustreError: 62258:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2121.984931] Lustre: server umount lustre-MDT0001 complete [ 2125.959668] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2131.224586] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2140.130472] Lustre: 60463:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775217965/real 1775217965] req@000000003927a488 x1861450834420224/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1775217981 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2140.155899] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2141.273732] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2141.919153] Lustre: 60464:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775217967/real 1775217967] req@000000008238c8fa x1861450834420352/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1775217983 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2145.247193] Lustre: 60462:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775217970/real 1775217970] req@0000000021e835e9 x1861450834420480/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1775217986 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2146.660205] LustreError: 63162:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2146.685553] LustreError: 63162:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2146.806215] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2146.981864] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2147.749928] Lustre: 60461:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775217972/real 1775217972] req@0000000078d483f6 x1861450834420736/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1775217988 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2149.868613] LustreError: 63163:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2149.881361] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2150.945968] LustreError: 64969:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2150.960749] LustreError: 64969:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2151.821473] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 2151.948548] Lustre: 60463:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775217977/real 1775217977] req@00000000960021df x1861450834420864/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1775217993 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2156.007184] LustreError: 64967:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2156.031423] LustreError: 64967:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2161.130823] LustreError: 64970:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2161.150645] LustreError: 64970:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2161.496888] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2162.032765] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2166.945914] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 2167.277992] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2172.767886] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2177.066315] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2187.751301] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2187.769912] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2187.771564] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2187.779744] Lustre: Skipped 1 previous similar message [ 2191.340404] Lustre: server umount lustre-MDT0000 complete [ 2192.874382] LustreError: 65658:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2192.897091] LustreError: 65658:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2196.261851] LustreError: 62258:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775218037 with bad export cookie 770550640034836263 [ 2196.266585] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2196.283427] LustreError: 62258:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2196.940684] Lustre: server umount lustre-MDT0001 complete [ 2204.872897] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2214.373428] Lustre: 60462:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775218039/real 1775218039] req@00000000f8b0b88a x1861450834448000/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1775218055 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2214.433222] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2214.448704] Lustre: Skipped 1 previous similar message [ 2215.325250] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2221.980221] LustreError: 63162:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2222.175884] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2222.254074] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2228.165723] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 2239.242140] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2243.699463] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2243.836409] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 2249.353660] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2254.325830] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2264.544956] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2264.548611] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2264.564976] Lustre: Skipped 1 previous similar message [ 2264.580954] Lustre: Skipped 4 previous similar messages [ 2269.665670] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2269.683160] Lustre: Skipped 2 previous similar messages [ 2276.319341] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2277.025412] Lustre: server umount lustre-MDT0000 complete [ 2279.906605] LustreError: 63162:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2279.918476] LustreError: 63162:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 10 previous similar messages [ 2280.806697] LustreError: 61531:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775218121 with bad export cookie 770550640034837642 [ 2280.808687] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2280.818567] LustreError: 61531:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2281.309337] Lustre: server umount lustre-MDT0001 complete [ 2289.660149] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2293.085608] Lustre: DEBUG MARKER: files compared the same [ 2306.229718] Lustre: server umount lustre-OST0000 complete [ 2325.853647] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing unload_modules_local [ 2329.149394] Key type lgssc unregistered [ 2329.495980] LNet: 70466:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2329.512765] LNetError: 70466:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2330.534245] LNet: Removed LNI 192.168.202.141@tcp [ 2331.410428] Key type .llcrypt unregistered [ 2331.416094] Key type ._llcrypt unregistered [ 2357.936673] Key type ._llcrypt registered [ 2357.941598] Key type .llcrypt registered [ 2358.112376] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 2359.613679] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2359.688772] alg: No test for adler32 (adler32-zlib) [ 2360.801173] Lustre: Lustre: Build Version: 2.17.51_74_g8e7ba6b [ 2360.992695] LNet: Added LNI 192.168.202.141@tcp [8/256/0/180] [ 2362.681358] Key type lgssc registered [ 2364.544275] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2372.480686] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2372.497613] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2374.123243] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2378.504522] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug +malloc all [ 2384.360964] LustreError: 71889:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2384.375473] LustreError: 71889:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2385.378840] LustreError: 71890:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2387.706453] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2388.183626] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2393.570211] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug +malloc all [ 2399.374608] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2403.305272] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2410.091149] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2410.363281] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2417.732618] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug +malloc all [ 2420.611602] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2424.871715] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2436.065987] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2436.068096] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2436.069330] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2436.078075] Lustre: Skipped 1 previous similar message [ 2441.202597] Lustre: server umount lustre-OST0000 complete [ 2446.307583] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2446.312627] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2446.327366] Lustre: Skipped 1 previous similar message [ 2446.330990] Lustre: Skipped 1 previous similar message [ 2451.434693] LustreError: 71884:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2451.450242] LustreError: 71884:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2451.542379] Lustre: server umount lustre-MDT0000 complete [ 2454.755081] LustreError: 73506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775218295 with bad export cookie 13207605462914177976 [ 2454.765570] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2454.771908] LustreError: 73506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2455.404457] Lustre: server umount lustre-MDT0001 complete [ 2465.729603] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing unload_modules_local [ 2468.122161] Key type lgssc unregistered [ 2468.374778] LNet: 75069:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2468.379176] LNetError: 75069:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2468.396882] LNet: Removed LNI 192.168.202.141@tcp [ 2469.141540] Key type .llcrypt unregistered [ 2469.144878] Key type ._llcrypt unregistered [ 2492.073859] Key type ._llcrypt registered [ 2492.077758] Key type .llcrypt registered [ 2492.239849] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 2493.459811] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2493.572694] alg: No test for adler32 (adler32-zlib) [ 2494.614472] Lustre: Lustre: Build Version: 2.17.51_74_g8e7ba6b [ 2494.789558] LNet: Added LNI 192.168.202.141@tcp [8/256/0/180] [ 2496.455633] Key type lgssc registered [ 2497.970619] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2506.417362] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2506.443630] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2522.079567] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2527.199277] LustreError: 76444:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.141@tcp: failed processing log, type 4: rc = -110 [ 2553.951903] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2561.040424] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 2567.323933] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2575.077733] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2575.162877] Lustre: *** cfs_fail_loc=706, val=0*** [ 2575.539879] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2575.603505] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2579.896556] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 2580.985740] LustreError: 76469:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2581.011245] LustreError: 76469:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2586.098295] LustreError: 76470:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2588.152693] LustreError: 77417:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2588.174964] LustreError: 77417:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2589.565182] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2590.261589] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2595.367537] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 2601.682747] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2606.955586] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2610.667259] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2610.682479] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2615.820857] Lustre: server umount lustre-OST0000 complete [ 2620.900688] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2620.924608] Lustre: Skipped 1 previous similar message [ 2620.938811] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2620.945927] Lustre: Skipped 1 previous similar message [ 2625.548515] Lustre: server umount lustre-MDT0000 complete [ 2626.016557] LustreError: 77421:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2626.042258] LustreError: 77421:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2629.249070] LustreError: 76451:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775218470 with bad export cookie 17816422456367062081 [ 2629.255793] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2629.261159] LustreError: 76451:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2629.594425] Lustre: server umount lustre-MDT0001 complete [ 2641.364337] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing unload_modules_local [ 2644.619441] Key type lgssc unregistered [ 2645.018598] LNet: 79707:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2645.038269] LNetError: 79707:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2645.057771] LNet: Removed LNI 192.168.202.141@tcp [ 2646.063519] Key type .llcrypt unregistered [ 2646.065180] Key type ._llcrypt unregistered [ 2671.229923] Key type ._llcrypt registered [ 2671.237624] Key type .llcrypt registered [ 2671.392946] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 2672.803762] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2672.842512] alg: No test for adler32 (adler32-zlib) [ 2673.902643] Lustre: Lustre: Build Version: 2.17.51_74_g8e7ba6b [ 2674.234237] LNet: Added LNI 192.168.202.141@tcp [8/256/0/180] [ 2675.919280] Key type lgssc registered [ 2677.343338] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2684.935916] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2684.968240] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2690.804945] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 2698.579100] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2699.009963] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2703.450744] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 2704.364664] LustreError: 81787:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2704.378839] LustreError: 81787:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2709.475886] LustreError: 81791:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2709.496922] LustreError: 81791:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2712.225820] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2712.531257] LustreError: 81790:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2712.552638] LustreError: 81790:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2712.585799] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2717.668575] LustreError: 81786:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2717.692549] LustreError: 81786:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2719.793079] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 2722.788181] LustreError: 81787:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2722.798641] LustreError: 81787:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2728.187802] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2728.207294] Lustre: MGS: Client df17ec76-4a0c-41c9-8f2c-1d82ce1aefb7 (at 0@lo) reconnecting [ 2728.238157] LustreError: 81801:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2728.497486] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2728.640726] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2732.673033] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 2748.904444] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2748.907310] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2748.914893] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2752.976993] Lustre: server umount lustre-OST0000 complete [ 2759.136547] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2759.141188] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2759.143259] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2759.143282] Lustre: Skipped 1 previous similar message [ 2759.192756] Lustre: Skipped 2 previous similar messages [ 2763.346140] Lustre: server umount lustre-MDT0000 complete [ 2764.259131] LustreError: 81790:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2764.286032] LustreError: 81790:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 2768.591676] Lustre: server umount lustre-MDT0001 complete [ 2772.729031] Lustre: server umount MGS complete [ 2783.643915] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 08:17:03 (1775218623) [ 2815.531184] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_hostid [ 2825.443209] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 2862.609883] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2867.777991] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 2876.929165] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2877.135637] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2877.185747] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2877.247765] Lustre: lustre-MDT0001: new disk, initializing [ 2877.306679] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2877.321104] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2877.325432] Lustre: 88872:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2877.329963] LustreError: 88872:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2880.602909] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 2891.290476] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2891.489568] Lustre: lustre-OST0000: new disk, initializing [ 2891.492705] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2891.580946] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2896.231522] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 2901.024513] Lustre: 90117:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@00000000363530a6 x1861451529625344/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1775218758 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2904.064994] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2910.468299] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2910.480743] Lustre: MGS: Client 8c4b4233-3946-493e-b020-dc11d3db679f (at 0@lo) reconnecting [ 2910.503679] LustreError: 88858:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2910.536076] Lustre: 88158:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2910.572189] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2910.647615] Lustre: lustre-MDT0000: new disk, initializing [ 2910.878233] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2910.917598] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2911.035986] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2911.970163] Lustre: 90578:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@00000000e18194f8 x1861451529631488/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1775218769 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2915.435507] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 2917.094139] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2917.105851] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2917.175250] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2941.921611] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2941.927448] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2941.952307] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2941.958927] Lustre: Skipped 1 previous similar message [ 2947.505480] Lustre: server umount lustre-OST0000 complete [ 2953.187436] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2953.187745] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2953.219469] Lustre: Skipped 1 previous similar message [ 2953.221198] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2953.236017] LustreError: Skipped 1 previous similar message [ 2953.265640] Lustre: Skipped 3 previous similar messages [ 2958.085376] Lustre: server umount lustre-MDT0000 complete [ 2958.316670] LustreError: 88843:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2958.360120] LustreError: 88843:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2969.197916] Lustre: server umount lustre-MDT0001 complete [ 2973.649993] Lustre: server umount MGS complete [ 2985.915222] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 08:20:25 (1775218825) [ 3003.089667] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing unload_modules_local [ 3006.316782] Key type lgssc unregistered [ 3006.743726] LNet: 93045:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3006.757831] LNetError: 93045:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3006.776187] LNet: Removed LNI 192.168.202.141@tcp [ 3007.967510] Key type .llcrypt unregistered [ 3007.972640] Key type ._llcrypt unregistered [ 3028.219242] Key type ._llcrypt registered [ 3028.222428] Key type .llcrypt registered [ 3028.321671] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 3029.209276] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3029.395332] alg: No test for adler32 (adler32-zlib) [ 3030.378918] Lustre: Lustre: Build Version: 2.17.51_74_g8e7ba6b [ 3030.656474] LNet: Added LNI 192.168.202.141@tcp [8/256/0/180] [ 3032.383331] Key type lgssc registered [ 3033.754288] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3041.463698] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3041.499653] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3043.161476] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3047.171595] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 3053.569772] LustreError: 94251:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3053.613247] LustreError: 94251:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3058.283490] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3058.710358] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3064.579481] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 3070.357417] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3073.702346] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3081.457980] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3081.939111] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3087.193613] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 3089.153062] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 3092.420324] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3104.746260] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3104.750389] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3104.759799] Lustre: Skipped 1 previous similar message [ 3104.762700] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3104.779413] Lustre: Skipped 1 previous similar message [ 3106.905527] Lustre: server umount lustre-OST0000 complete [ 3114.982773] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3114.997211] Lustre: Skipped 1 previous similar message [ 3115.001835] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3118.015238] Lustre: server umount lustre-MDT0000 complete [ 3120.097796] LustreError: 95145:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3120.111352] LustreError: 95145:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3122.178151] LustreError: 94233:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775218963 with bad export cookie 16838564864237202865 [ 3122.191819] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3122.664643] Lustre: server umount lustre-MDT0001 complete [ 3136.433957] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing unload_modules_local [ 3139.452659] Key type lgssc unregistered [ 3139.872923] LNet: 97431:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3139.880809] LNetError: 97431:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3139.903848] LNet: Removed LNI 192.168.202.141@tcp [ 3140.977639] Key type .llcrypt unregistered [ 3140.983068] Key type ._llcrypt unregistered [ 3159.320739] Key type ._llcrypt registered [ 3159.325246] Key type .llcrypt registered [ 3159.440412] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 3160.376135] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3160.461741] alg: No test for adler32 (adler32-zlib) [ 3161.751242] Lustre: Lustre: Build Version: 2.17.51_74_g8e7ba6b [ 3162.059403] LNet: Added LNI 192.168.202.141@tcp [8/256/0/180] [ 3163.871462] Key type lgssc registered [ 3165.461758] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3172.331972] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3172.343443] LustreError: 98486:0:(tgt_mount.c:2501:server_fill_super()) cfs_race id 716 sleeping [ 3172.400414] LustreError: 98498:0:(tgt_mount.c:2501:server_fill_super()) cfs_fail_race id 716 waking [ 3172.406520] LustreError: 98486:0:(tgt_mount.c:2501:server_fill_super()) cfs_fail_race id 716 awake: rc=4952 [ 3172.415197] LustreError: 98486:0:(tgt_mount.c:2513:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3172.431534] LustreError: 98486:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3172.433316] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3174.566672] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3182.600292] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3183.172344] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3188.891885] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 3193.816520] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing lsmod [ 3196.690239] LustreError: 99666:0:(tgt_mount.c:2501:server_fill_super()) cfs_race id 716 sleeping [ 3196.702722] LustreError: 99669:0:(tgt_mount.c:2501:server_fill_super()) cfs_fail_race id 716 waking [ 3196.706713] LustreError: 99666:0:(tgt_mount.c:2501:server_fill_super()) cfs_fail_race id 716 awake: rc=4995 [ 3196.714622] LustreError: 99666:0:(tgt_mount.c:2513:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3196.717604] LustreError: 99666:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3196.741533] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3197.124346] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3199.249649] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3203.552472] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3203.557596] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3203.572331] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3204.581625] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3204.583057] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3204.595256] Lustre: Skipped 1 previous similar message [ 3204.607578] Lustre: Skipped 1 previous similar message [ 3205.994973] Lustre: server umount lustre-MDT0000 complete [ 3208.744141] LustreError: 98536:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3208.773145] LustreError: 98536:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 3209.947268] LustreError: 98515:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775219051 with bad export cookie 17688082203463520373 [ 3209.965300] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3209.966989] LustreError: 98515:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3210.847299] LustreError: 98967:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000be24cc08 x1861452041311232/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3211.318264] Lustre: server umount lustre-MDT0001 complete [ 3227.042237] Lustre: server umount lustre-OST0000 complete [ 3237.355393] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3237.849830] LustreError: 100736:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3237.936076] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3242.977909] LustreError: 100737:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3243.959574] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 3247.079166] LustreError: 100736:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3252.193971] LustreError: 100737:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3254.760186] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3255.268749] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3259.954361] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 3265.759834] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3269.537320] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3276.750714] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3277.249867] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3282.363501] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:97) [ 3284.665536] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 3291.542557] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3300.833480] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3300.842156] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3300.855699] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3302.891214] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3302.909376] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3302.935938] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3305.033717] Lustre: server umount lustre-OST0000 complete [ 3311.589927] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3311.602717] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3311.633120] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3311.640884] Lustre: Skipped 1 previous similar message [ 3316.023527] Lustre: server umount lustre-MDT0000 complete [ 3318.241037] LustreError: 100731:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3318.257359] LustreError: 100731:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3319.773265] LustreError: 102900:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775219160 with bad export cookie 17688082203463521304 [ 3319.776597] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3319.788892] LustreError: 102900:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3320.038206] Lustre: server umount lustre-MDT0001 complete [ 3331.949966] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing unload_modules_local [ 3334.923941] Key type lgssc unregistered [ 3335.299826] LNet: 103926:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3335.317688] LNetError: 103926:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3335.346238] LNet: Removed LNI 192.168.202.141@tcp [ 3336.244209] Key type .llcrypt unregistered [ 3336.245654] Key type ._llcrypt unregistered [ 3360.344519] Key type ._llcrypt registered [ 3360.347063] Key type .llcrypt registered [ 3360.449525] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 3361.436840] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3361.521810] alg: No test for adler32 (adler32-zlib) [ 3362.722829] Lustre: Lustre: Build Version: 2.17.51_74_g8e7ba6b [ 3362.935886] LNet: Added LNI 192.168.202.141@tcp [8/256/0/180] [ 3364.639157] Key type lgssc registered [ 3365.757651] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3373.644007] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3373.665944] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3375.158949] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3380.122790] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 3385.316332] LustreError: 105337:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3385.338094] LustreError: 105337:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3388.386221] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3388.831407] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3394.064617] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 3400.351172] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3404.602676] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3412.398726] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3412.802394] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3419.749827] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 3419.958143] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3425.665056] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3431.914404] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3436.367833] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3444.711229] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3444.720762] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3444.745788] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3445.728276] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3445.732426] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3445.740695] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3445.768316] Lustre: Skipped 1 previous similar message [ 3447.880683] Lustre: server umount lustre-OST0000 complete [ 3455.457930] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3455.463416] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3455.482083] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3458.925837] Lustre: server umount lustre-MDT0000 complete [ 3461.102024] LustreError: 106284:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3461.130456] LustreError: 106284:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3463.160633] LustreError: 105317:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775219304 with bad export cookie 7753494912054923666 [ 3463.171638] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3463.176057] LustreError: 105317:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3463.642249] Lustre: server umount lustre-MDT0001 complete [ 3475.825981] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing unload_modules_local [ 3478.831874] Key type lgssc unregistered [ 3479.134709] LNet: 108625:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3479.146183] LNetError: 108625:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3479.161973] LNet: Removed LNI 192.168.202.141@tcp [ 3479.804794] Key type .llcrypt unregistered [ 3479.807679] Key type ._llcrypt unregistered [ 3497.804306] Key type ._llcrypt registered [ 3497.811023] Key type .llcrypt registered [ 3497.952990] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 3499.675239] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3499.698100] alg: No test for adler32 (adler32-zlib) [ 3500.820611] Lustre: Lustre: Build Version: 2.17.51_74_g8e7ba6b [ 3501.141273] LNet: Added LNI 192.168.202.141@tcp [8/256/0/180] [ 3502.935168] Key type lgssc registered [ 3504.024828] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3510.729934] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3510.765421] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3512.240884] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3516.320635] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 3522.531408] LustreError: 109830:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3522.548602] LustreError: 109830:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3524.332473] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3524.722460] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3529.006210] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 3535.493851] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3539.881772] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3547.658130] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3547.929334] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3554.371475] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 3557.198411] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3560.385772] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3569.637989] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3569.643769] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3569.653888] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3572.708089] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3572.712252] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3572.729666] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3575.955744] Lustre: server umount lustre-OST0000 complete [ 3580.903925] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3580.918615] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3580.947029] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3580.950014] Lustre: Skipped 1 previous similar message [ 3582.946759] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3582.949091] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3582.979554] Lustre: Skipped 1 previous similar message [ 3586.111736] Lustre: server umount lustre-MDT0000 complete [ 3588.078482] LustreError: 109825:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3588.103640] LustreError: 109825:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3590.281607] LustreError: 109809:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775219431 with bad export cookie 874343051189495083 [ 3590.288574] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3590.293335] LustreError: 109809:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3590.601986] Lustre: server umount lustre-MDT0001 complete [ 3603.278321] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing unload_modules_local [ 3606.604718] Key type lgssc unregistered [ 3607.068605] LNet: 113008:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3607.080254] LNetError: 113008:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3607.115823] LNet: Removed LNI 192.168.202.141@tcp [ 3608.139328] Key type .llcrypt unregistered [ 3608.141840] Key type ._llcrypt unregistered [ 3634.759495] Key type ._llcrypt registered [ 3634.764266] Key type .llcrypt registered [ 3634.898277] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 3636.494228] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3636.532594] alg: No test for adler32 (adler32-zlib) [ 3637.664467] Lustre: Lustre: Build Version: 2.17.51_74_g8e7ba6b [ 3637.973708] LNet: Added LNI 192.168.202.141@tcp [8/256/0/180] [ 3639.680869] Key type lgssc registered [ 3641.264406] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3650.069642] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3650.086181] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3651.562504] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3656.696686] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 3661.796838] LustreError: 114412:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3661.821721] LustreError: 114412:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3666.892384] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3666.914078] LustreError: 114411:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3667.270138] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3672.518189] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 3679.295509] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3683.621514] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3692.218079] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3692.719547] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3696.907623] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:163 to 0x280000400:193) [ 3701.081718] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 3707.939530] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3719.503229] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3723.851592] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3723.856915] Lustre: Skipped 2 previous similar messages [ 3728.435708] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3728.476939] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3728.740638] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3728.750504] Lustre: Skipped 2 previous similar messages [ 3738.150253] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3738.155321] Lustre: Skipped 1 previous similar message [ 3743.914481] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.202.41@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3743.923824] Lustre: Skipped 2 previous similar messages [ 3744.111791] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.202.41@tcp 192.168.0.[2,10]@tcp [ 3754.017966] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.202.41/24@tcp in log lustre-MDT0000 [ 3754.026711] Lustre: Skipped 2 previous similar messages [ 3761.761913] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.41/24@tcp [ 3761.776227] Lustre: Skipped 1 previous similar message [ 3773.944614] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3773.946921] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3773.965105] LustreError: Skipped 1 previous similar message [ 3773.973083] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3773.992147] Lustre: Skipped 1 previous similar message [ 3775.468690] Lustre: server umount lustre-OST0000 complete [ 3780.068312] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3780.078224] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3780.112080] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3780.124370] Lustre: Skipped 1 previous similar message [ 3784.160674] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3784.162034] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3785.782662] Lustre: server umount lustre-MDT0000 complete [ 3789.186595] LustreError: 114392:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775219630 with bad export cookie 15037518703315451132 [ 3789.188471] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3789.192614] LustreError: 114392:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3789.471542] Lustre: server umount lustre-MDT0001 complete [ 3804.611332] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing unload_modules_local [ 3809.331820] Key type lgssc unregistered [ 3809.640684] LNet: 119056:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3809.645389] LNetError: 119056:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3809.666065] LNet: Removed LNI 192.168.202.141@tcp [ 3810.856496] Key type .llcrypt unregistered [ 3810.859512] Key type ._llcrypt unregistered [ 3846.334874] Key type ._llcrypt registered [ 3846.344816] Key type .llcrypt registered [ 3846.490253] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 3847.425104] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3847.482327] alg: No test for adler32 (adler32-zlib) [ 3848.879913] Lustre: Lustre: Build Version: 2.17.51_74_g8e7ba6b [ 3849.179791] LNet: Added LNI 192.168.202.141@tcp [8/256/0/180] [ 3850.897416] Key type lgssc registered [ 3852.271581] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3861.032514] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3861.071825] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3862.757165] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3862.795075] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.202.41/24@tcp [ 3862.847281] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3867.522913] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 3872.232398] LustreError: 120601:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3872.246756] LustreError: 120601:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3877.110330] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3877.346857] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 3877.556895] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3877.606574] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.41/24@tcp [ 3877.757110] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3883.066338] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 3889.393751] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3893.604700] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3901.008699] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3901.397477] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3907.595141] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:201 to 0x280000400:225) [ 3908.730506] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 3915.500370] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3928.032917] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3928.043378] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3928.057159] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3928.063572] Lustre: Skipped 1 previous similar message [ 3932.128100] Lustre: server umount lustre-OST0000 complete [ 3938.273814] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3938.325234] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3938.334408] Lustre: Skipped 2 previous similar messages [ 3942.698661] Lustre: server umount lustre-MDT0000 complete [ 3943.394653] LustreError: 120600:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3943.410978] LustreError: 120600:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3946.893180] LustreError: 120581:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775219787 with bad export cookie 9816664591251918891 [ 3946.895630] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3946.914062] LustreError: 120581:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3947.521497] Lustre: server umount lustre-MDT0001 complete [ 3959.389539] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing unload_modules_local [ 3962.490260] Key type lgssc unregistered [ 3962.810318] LNet: 123837:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3962.819049] LNetError: 123837:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3962.840121] LNet: Removed LNI 192.168.202.141@tcp [ 3963.800208] Key type .llcrypt unregistered [ 3963.803330] Key type ._llcrypt unregistered [ 3997.507436] Key type ._llcrypt registered [ 3997.510291] Key type .llcrypt registered [ 3997.648330] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_hostid [ 4013.698547] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 4015.180208] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4015.212765] alg: No test for adler32 (adler32-zlib) [ 4016.386123] Lustre: Lustre: Build Version: 2.17.51_74_g8e7ba6b [ 4016.716142] LNet: Added LNI 192.168.202.141@tcp [8/256/0/180] [ 4018.503148] Key type lgssc registered [ 4019.639642] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4057.076489] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4057.122493] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4058.522783] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4058.576587] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4058.733752] Lustre: lustre-MDT0000: new disk, initializing [ 4058.865226] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4058.927228] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4065.072682] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 4076.871230] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4077.016256] Lustre: 127207:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4077.096132] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4077.098894] Lustre: Skipped 1 previous similar message [ 4077.204264] Lustre: lustre-MDT0001: new disk, initializing [ 4077.310255] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4077.355056] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4077.375472] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4082.330588] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 4090.308684] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4094.624315] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4102.203756] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4102.513906] Lustre: lustre-OST0000: new disk, initializing [ 4102.517865] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4102.594408] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4109.061898] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 4112.456147] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4112.468242] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4112.556894] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4116.973141] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4121.766536] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4121.948210] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4126.401065] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4126.619214] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4133.345820] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4133.359940] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4133.368531] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4133.379881] Lustre: Skipped 1 previous similar message [ 4133.406365] Lustre: Skipped 1 previous similar message [ 4134.718446] Lustre: server umount lustre-OST0000 complete [ 4143.584636] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4143.585468] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4143.598991] Lustre: Skipped 1 previous similar message [ 4145.495450] Lustre: server umount lustre-MDT0000 complete [ 4149.945246] LustreError: 127200:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775219991 with bad export cookie 15895526983184788733 [ 4149.966951] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4150.480381] Lustre: server umount lustre-MDT0001 complete [ 4160.696409] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4161.439845] LustreError: 131018:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4161.467740] LustreError: 131018:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4161.529647] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4166.440735] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 4166.624673] LustreError: 131019:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4171.744774] LustreError: 131018:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4175.842529] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4176.176753] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4180.920839] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 4186.430583] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4190.443612] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4196.862661] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4197.338911] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4203.935626] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 4209.886719] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4215.724045] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4215.927781] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4221.371305] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4221.661588] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4231.184780] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4231.376551] Lustre: lustre-OST0001: new disk, initializing [ 4231.379223] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4231.466070] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4237.816909] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 4239.621862] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4239.642827] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4239.736696] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4247.258422] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4247.597565] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4253.170934] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4253.462306] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4258.164694] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8f114b959800.ost_server_uuid 50 [ 4259.810445] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8f114b959800.ost_server_uuid in FULL state after 0 sec [ 4270.563103] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4270.571093] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4270.572882] LustreError: Skipped 1 previous similar message [ 4270.584535] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4270.589018] Lustre: Skipped 1 previous similar message [ 4270.615793] Lustre: Skipped 2 previous similar messages [ 4272.107165] Lustre: server umount lustre-OST0001 complete [ 4275.684610] LustreError: 132614:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4275.704159] LustreError: 132614:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4278.247051] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4278.251243] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4278.252545] LustreError: Skipped 2 previous similar messages [ 4278.253762] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4278.289064] Lustre: Skipped 1 previous similar message [ 4280.805712] LustreError: 135174:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4280.837742] LustreError: 135174:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4282.572795] Lustre: server umount lustre-OST0000 complete [ 4288.994320] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4289.017284] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4289.039589] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4289.047427] Lustre: Skipped 3 previous similar messages [ 4301.279161] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4301.287909] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4301.297605] Lustre: Skipped 5 previous similar messages [ 4301.576368] Lustre: server umount lustre-MDT0000 complete [ 4305.303199] LustreError: 131000:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775220146 with bad export cookie 15895526983184789615 [ 4305.308402] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4305.317814] LustreError: 131000:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 4305.690830] Lustre: server umount lustre-MDT0001 complete [ 4325.046754] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing unload_modules_local [ 4327.803757] Key type lgssc unregistered [ 4328.151765] LNet: 136858:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4328.162454] LNetError: 136858:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4328.185615] LNet: Removed LNI 192.168.202.141@tcp [ 4329.211080] Key type .llcrypt unregistered [ 4329.215322] Key type ._llcrypt unregistered [ 4365.004988] Key type ._llcrypt registered [ 4365.007274] Key type .llcrypt registered [ 4365.097502] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_hostid [ 4379.487437] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 4380.230256] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4380.304678] alg: No test for adler32 (adler32-zlib) [ 4381.489142] Lustre: Lustre: Build Version: 2.17.51_74_g8e7ba6b [ 4381.708485] LNet: Added LNI 192.168.202.141@tcp [8/256/0/180] [ 4383.359159] Key type lgssc registered [ 4384.327522] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4414.587825] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4414.620022] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4415.993620] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4416.042892] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4416.124864] Lustre: lustre-MDT0000: new disk, initializing [ 4416.224525] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4416.241206] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4420.504818] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 4429.775618] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4429.853810] Lustre: 140999:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4429.930419] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4429.937555] Lustre: Skipped 1 previous similar message [ 4430.001701] Lustre: lustre-MDT0001: new disk, initializing [ 4430.068421] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4430.100285] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4430.108489] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4433.481356] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 4440.254841] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4443.895280] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4449.765311] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4449.970104] Lustre: lustre-OST0000: new disk, initializing [ 4449.977872] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4450.039638] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4455.513788] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 4458.058404] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4458.072036] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4458.211588] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4461.395108] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4465.843725] Lustre: Failing over lustre-OST0000 [ 4466.010823] Lustre: server umount lustre-OST0000 complete [ 4482.357345] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4482.562336] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4482.597583] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4482.611977] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4484.511068] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4484.575265] Lustre: 139221:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775220309/real 1775220309] req@00000000aad29773 x1861453319536128/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1775220325 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4487.032394] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 4487.682889] Lustre: lustre-OST0000: Recovery over after 0:03, of 3 clients 3 recovered and 0 were evicted. [ 4487.685826] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4489.763316] Lustre: 139222:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775220314/real 1775220314] req@00000000d6ea20f4 x1861453319537280/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1775220330 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4489.790035] Lustre: 139222:0:(client.c:2479:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 4490.065754] Lustre: Failing over lustre-MDT0000 [ 4490.203813] Lustre: server umount lustre-MDT0000 complete [ 4491.103245] Lustre: 139221:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775220314/real 1775220314] req@000000005cfc3334 x1861453319537152/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1775220330 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4491.744092] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4491.749739] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4491.755911] Lustre: Skipped 1 previous similar message [ 4492.772825] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4495.839779] Lustre: 139223:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775220320/real 1775220320] req@00000000a58e96e0 x1861453319538304/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1775220336 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4497.890936] LustreError: 143059:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4497.905638] LustreError: 143059:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 4499.842765] LustreError: 141006:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.41@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4503.015349] LustreError: 141010:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4503.023336] LustreError: 141010:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4506.616746] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4506.715792] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4506.728782] Lustre: 141002:0:(mgc_request.c:1917:mgc_process_log()) MGC192.168.202.141@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 4507.006053] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4507.063497] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4510.086559] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4510.417721] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 4512.230038] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4512.234026] Lustre: Skipped 1 previous similar message [ 4512.257827] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 4512.284642] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4518.369219] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4518.372586] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4518.400340] Lustre: Skipped 1 previous similar message [ 4518.404788] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4519.804214] Lustre: server umount lustre-OST0000 complete [ 4527.585047] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4527.589111] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4527.589972] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4527.596402] Lustre: Skipped 1 previous similar message [ 4532.705227] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4532.707975] Lustre: Skipped 1 previous similar message [ 4537.316959] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4537.524053] Lustre: server umount lustre-MDT0000 complete [ 4537.825684] LustreError: 143059:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4537.846085] LustreError: 143059:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4540.968886] LustreError: 140990:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775220382 with bad export cookie 12146868480787937504 [ 4540.972872] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4540.975753] LustreError: 140990:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4541.303503] Lustre: server umount lustre-MDT0001 complete [ 4550.374670] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing unload_modules_local [ 4552.633913] Key type lgssc unregistered [ 4552.906724] LNet: 146234:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4552.916891] LNetError: 146234:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4552.936145] LNet: Removed LNI 192.168.202.141@tcp [ 4553.632102] Key type .llcrypt unregistered [ 4553.633313] Key type ._llcrypt unregistered [ 4586.812292] Key type ._llcrypt registered [ 4586.814347] Key type .llcrypt registered [ 4586.920439] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 4587.834638] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4587.876160] alg: No test for adler32 (adler32-zlib) [ 4588.996982] Lustre: Lustre: Build Version: 2.17.51_74_g8e7ba6b [ 4589.250240] LNet: Added LNI 192.168.202.141@tcp [8/256/0/180] [ 4590.919220] Key type lgssc registered [ 4592.022859] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4598.482568] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4598.527309] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4600.057710] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4603.474693] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 4610.369895] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4610.533382] LustreError: 147736:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4610.547053] LustreError: 147736:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4614.095654] LustreError: 147765:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4614.106157] LustreError: 147765:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4614.111745] LustreError: 147765:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 14, retries 0, failed: rc = -5 [ 4614.404720] Lustre: server umount lustre-MDT0000 complete [ 4627.941856] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_hostid [ 4635.499875] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 4664.230840] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4664.418233] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4664.436955] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4664.519620] Lustre: lustre-MDT0000: new disk, initializing [ 4664.583054] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4664.596172] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4668.137471] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 4675.832633] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4675.885161] Lustre: 150918:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4675.923621] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4675.927357] Lustre: Skipped 1 previous similar message [ 4675.986815] Lustre: lustre-MDT0001: new disk, initializing [ 4676.058328] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4676.098393] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4676.105140] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4679.311531] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 4685.106163] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4688.161130] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4693.642499] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4693.853419] Lustre: lustre-OST0000: new disk, initializing [ 4693.857815] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4693.921249] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4697.933505] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 4699.682394] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4699.689778] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4699.800647] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4704.071329] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4707.992260] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4709.128244] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 08:49:09 (1775220549) [ 4879.327875] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4879.334345] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4879.338982] Lustre: Skipped 1 previous similar message [ 4884.457453] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4884.463566] Lustre: Skipped 4 previous similar messages [ 4889.572730] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4889.577563] Lustre: Skipped 2 previous similar messages [ 4892.127173] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4892.305254] Lustre: server umount lustre-MDT0000 complete [ 4894.688627] LustreError: 151769:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4894.713069] LustreError: 151769:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4894.827725] LustreError: 152660:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775220735 with bad export cookie 3333580099640805265 [ 4894.828216] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4894.835371] LustreError: 152660:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4899.820460] LustreError: 150926:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4899.820669] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4899.841349] Lustre: Skipped 1 previous similar message [ 4899.844412] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4901.133239] Lustre: server umount lustre-MDT0001 complete [ 4918.239247] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4918.336872] Lustre: server umount lustre-OST0000 complete [ 4923.939531] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_hostid [ 4929.814673] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 4953.593322] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4953.731944] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4953.757359] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4953.814549] Lustre: lustre-MDT0000: new disk, initializing [ 4953.884216] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4953.901574] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4956.801680] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 4964.619582] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4964.721389] Lustre: 157164:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4964.757619] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4964.760234] Lustre: Skipped 1 previous similar message [ 4964.820989] Lustre: lustre-MDT0001: new disk, initializing [ 4964.890744] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4964.932891] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4964.941963] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4968.041399] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 4973.743855] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4976.060204] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4980.657848] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4980.839204] Lustre: lustre-OST0000: new disk, initializing [ 4980.842893] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4980.901266] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4984.904785] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 4986.923111] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4986.935048] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4986.994524] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4989.921301] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4992.946313] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4993.197264] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4996.232774] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4996.376805] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5002.212618] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5002.220589] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5002.226238] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5002.239995] Lustre: Skipped 1 previous similar message [ 5003.910433] Lustre: server umount lustre-OST0000 complete [ 5007.328121] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5007.335369] Lustre: Skipped 2 previous similar messages [ 5012.449229] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5012.454136] Lustre: Skipped 3 previous similar messages [ 5013.216977] Lustre: server umount lustre-MDT0000 complete [ 5015.337997] LustreError: 157154:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775220856 with bad export cookie 3333580099640964095 [ 5015.345113] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5015.516900] Lustre: server umount lustre-MDT0001 complete [ 5020.512773] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 08:54:21 (1775220861) [ 5031.682835] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_hostid [ 5038.064499] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 5060.667790] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5060.844419] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5060.866714] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5060.920433] Lustre: lustre-MDT0000: new disk, initializing [ 5060.977731] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5060.988604] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5063.892570] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 5071.875782] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5071.992339] Lustre: 163476:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5072.207967] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5075.065874] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 5081.003500] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5083.960688] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5088.939346] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5093.295501] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 5097.030939] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5097.036000] Lustre: Skipped 1 previous similar message [ 5097.038657] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5097.130761] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5099.014218] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5107.686800] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5107.690910] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5107.698223] LustreError: Skipped 1 previous similar message [ 5107.704253] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5107.723598] Lustre: Skipped 1 previous similar message [ 5107.734719] Lustre: Skipped 1 previous similar message [ 5109.843695] Lustre: server umount lustre-OST0000 complete [ 5118.830863] Lustre: server umount lustre-MDT0000 complete [ 5121.504054] LustreError: 163467:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775220962 with bad export cookie 3333580099640964977 [ 5121.514455] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5121.519255] LustreError: 163467:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 5121.937920] Lustre: server umount lustre-MDT0001 complete [ 5129.112886] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 08:56:09 (1775220969) [ 5142.591590] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_hostid [ 5150.879227] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 5179.814828] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5180.073735] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5180.098899] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5180.102705] Lustre: Skipped 3 previous similar messages [ 5180.173161] Lustre: lustre-MDT0000: new disk, initializing [ 5180.177515] Lustre: Skipped 2 previous similar messages [ 5180.272558] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5180.280256] Lustre: Skipped 2 previous similar messages [ 5180.299529] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5183.882500] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 5192.479594] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5192.606570] Lustre: 169392:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5192.853377] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5196.116728] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 5202.286516] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5205.291723] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5210.996600] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5215.840602] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 5219.031824] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5223.091345] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5229.536576] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5229.541351] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5229.543973] LustreError: Skipped 1 previous similar message [ 5229.550995] Lustre: Skipped 2 previous similar messages [ 5229.553682] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5229.569482] Lustre: Skipped 5 previous similar messages [ 5235.077758] Lustre: server umount lustre-OST0000 complete [ 5238.753103] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5244.267754] Lustre: server umount lustre-MDT0000 complete [ 5244.896207] LustreError: 169405:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5244.903275] LustreError: 169405:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5246.225037] LustreError: 169385:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775221087 with bad export cookie 3333580099640966062 [ 5246.228708] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5246.233321] LustreError: 169385:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 5246.415276] Lustre: server umount lustre-MDT0001 complete [ 5252.947773] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing unload_modules_local [ 5254.449889] Key type lgssc unregistered [ 5254.664828] LNet: 172868:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5254.670466] LNetError: 172868:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5254.684744] LNet: Removed LNI 192.168.202.141@tcp [ 5255.208788] Key type .llcrypt unregistered [ 5255.210521] Key type ._llcrypt unregistered [ 5269.823189] Key type ._llcrypt registered [ 5269.825671] Key type .llcrypt registered [ 5269.892285] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 5270.646875] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5270.663816] alg: No test for adler32 (adler32-zlib) [ 5271.614664] Lustre: Lustre: Build Version: 2.17.51_74_g8e7ba6b [ 5271.780631] LNet: Added LNI 192.168.202.141@tcp [8/256/0/180] [ 5273.407183] Key type lgssc registered [ 5274.216527] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5279.373961] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5279.389870] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5280.718299] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5283.691201] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 5289.294642] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5289.553615] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5292.724739] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 5296.995379] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5300.292248] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5305.425522] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5305.677163] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5306.744598] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5309.933505] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 5313.593929] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5330.400381] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5330.405139] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5330.406867] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5330.411897] Lustre: Skipped 1 previous similar message [ 5330.417193] Lustre: Skipped 1 previous similar message [ 5335.293508] Lustre: server umount lustre-OST0000 complete [ 5340.645694] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5340.656470] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5340.659427] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5340.673916] Lustre: Skipped 1 previous similar message [ 5344.352530] Lustre: server umount lustre-MDT0000 complete [ 5345.759810] LustreError: 174956:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5345.776335] LustreError: 174956:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5346.969808] LustreError: 174226:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775221188 with bad export cookie 2704397440945805043 [ 5346.975918] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5346.979780] LustreError: 174226:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5347.270296] Lustre: server umount lustre-MDT0001 complete [ 5355.227488] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing unload_modules_local [ 5357.216467] Key type lgssc unregistered [ 5357.509870] LNet: 177446:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5357.513953] LNetError: 177446:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5357.529890] LNet: Removed LNI 192.168.202.141@tcp [ 5358.102968] Key type .llcrypt unregistered [ 5358.109313] Key type ._llcrypt unregistered [ 5373.979979] Key type ._llcrypt registered [ 5373.980960] Key type .llcrypt registered [ 5374.032703] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 5374.901914] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5374.924061] alg: No test for adler32 (adler32-zlib) [ 5375.931572] Lustre: Lustre: Build Version: 2.17.51_74_g8e7ba6b [ 5376.075686] LNet: Added LNI 192.168.202.141@tcp [8/256/0/180] [ 5377.695962] Key type lgssc registered [ 5378.555266] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5384.149916] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5384.167242] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5385.611966] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5388.436965] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 5394.650379] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5394.946038] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5397.818619] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 5401.405285] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5403.715194] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5408.409052] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5408.673200] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5411.763319] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5412.595017] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 5416.441352] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5427.172976] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5427.180023] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5427.192495] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5429.140084] Lustre: lustre-OST0000: Not available for connect from 192.168.202.41@tcp (stopping) [ 5429.224308] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5431.438133] Lustre: server umount lustre-OST0000 complete [ 5434.630448] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff8f1160bd6800.ost_server_uuid 50 [ 5435.634135] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5447.257148] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5447.272733] Lustre: Skipped 1 previous similar message [ 5447.273736] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5447.295561] Lustre: Skipped 3 previous similar messages [ 5449.315594] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5449.324446] Lustre: Skipped 1 previous similar message [ 5453.561665] Lustre: server umount lustre-MDT0000 complete [ 5454.836625] LustreError: 178826:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5454.848255] LustreError: 178826:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5455.821579] LustreError: 178807:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775221296 with bad export cookie 8449420419573526192 [ 5455.826185] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5455.827597] LustreError: 178807:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5456.046622] Lustre: server umount lustre-MDT0001 complete [ 5460.615770] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 09:01:41 (1775221301) [ 5464.455392] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5464.835589] LustreError: 182314:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5464.879381] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5467.164694] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 5470.177786] LustreError: 182315:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5472.574102] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5472.802757] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5475.294264] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 5478.572317] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5480.752084] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5484.984392] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5485.189762] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5488.864294] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 5489.322357] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5491.919161] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5496.369553] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5496.473689] Lustre: lustre-OST0001: new disk, initializing [ 5496.477476] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5499.794684] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 5504.207228] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5505.586236] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5505.592046] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5505.659844] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5508.065445] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5508.074768] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5508.082281] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5510.112660] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5520.261042] Lustre: lustre-OST0000: Not available for connect from 192.168.202.41@tcp (stopping) [ 5520.266048] Lustre: Skipped 6 previous similar messages [ 5520.351221] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5520.432748] Lustre: server umount lustre-OST0000 complete [ 5520.872685] LustreError: 184288:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5520.881020] LustreError: 184288:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5524.476725] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5524.662860] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5525.391493] LustreError: 183908:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.41@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5528.041589] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5528.213333] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5535.622996] LustreError: 183908:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.41@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5535.640508] LustreError: 183908:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 5541.346403] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5541.347956] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5541.350783] LustreError: Skipped 1 previous similar message [ 5541.351808] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5541.360864] Lustre: Skipped 2 previous similar messages [ 5541.367542] Lustre: Skipped 1 previous similar message [ 5546.058900] Lustre: server umount lustre-OST0001 complete [ 5549.535869] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5549.541123] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5554.440585] Lustre: server umount lustre-MDT0000 complete [ 5556.706062] LustreError: 182315:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5556.721346] LustreError: 182315:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 5557.028971] LustreError: 183030:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775221398 with bad export cookie 8449420419573527249 [ 5557.036750] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5557.037940] LustreError: 183030:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5557.244743] Lustre: server umount lustre-MDT0001 complete [ 5571.711351] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 09:03:32 (1775221412) [ 5575.916349] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5576.081187] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5576.115935] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5576.220768] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5576.224783] Lustre: Skipped 1 previous similar message [ 5578.723271] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 5584.417588] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5584.503162] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5584.545603] Lustre: 188097:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5587.493116] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 5590.779022] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5593.336135] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5598.297037] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5598.420376] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5602.591593] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 5605.809614] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5608.045332] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 5610.118593] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5610.152415] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5610.267956] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5610.271986] Lustre: Skipped 2 previous similar messages [ 5613.796667] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 5617.073097] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5618.657243] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5618.666170] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5618.671690] Lustre: Skipped 1 previous similar message [ 5618.674275] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5618.677886] Lustre: Skipped 3 previous similar messages [ 5624.716867] Lustre: server umount lustre-OST0000 complete [ 5627.783026] LustreError: 189718:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.41@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5638.121323] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5642.679916] Lustre: server umount lustre-OST0001 complete [ 5651.144496] Lustre: server umount lustre-MDT0000 complete [ 5653.215048] LustreError: 188088:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775221494 with bad export cookie 8449420419573528656 [ 5653.218108] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5653.224917] LustreError: 188088:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5653.393255] Lustre: server umount lustre-MDT0001 complete [ 5665.744667] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 09:05:06 (1775221506) [ 5674.942611] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_hostid [ 5680.101180] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 5702.099250] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5702.254692] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5702.272524] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5702.328671] Lustre: lustre-MDT0000: new disk, initializing [ 5702.379509] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5702.396334] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5704.901157] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 5711.280696] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5711.340785] Lustre: 195680:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5711.374769] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5711.376696] Lustre: Skipped 1 previous similar message [ 5711.427597] Lustre: lustre-MDT0001: new disk, initializing [ 5711.504687] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5711.513119] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5713.682406] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 5718.128661] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5720.210582] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5723.823906] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5723.991405] Lustre: lustre-OST0000: new disk, initializing [ 5723.996575] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5727.144167] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5727.153993] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5727.232138] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5727.452333] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 5732.277414] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5734.540966] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5734.655816] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5736.584988] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5736.691034] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5742.562111] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5742.563808] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5742.567368] LustreError: Skipped 2 previous similar messages [ 5742.574063] Lustre: Skipped 5 previous similar messages [ 5742.575858] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5742.588514] Lustre: Skipped 9 previous similar messages [ 5743.786202] Lustre: server umount lustre-OST0000 complete [ 5751.746533] Lustre: server umount lustre-MDT0000 complete [ 5752.801087] LustreError: 196470:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5752.810550] LustreError: 196470:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 5753.479491] LustreError: 195671:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775221594 with bad export cookie 8449420419573530217 [ 5753.479894] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5753.486026] LustreError: 195671:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5753.638488] Lustre: server umount lustre-MDT0001 complete [ 5758.109924] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5760.229217] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 5764.657217] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5766.862402] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 5769.896475] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5772.076883] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5775.431550] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5778.226314] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 5780.707372] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5782.636476] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5784.801661] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 5786.611316] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5789.841393] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 3 sec [ 5796.829624] Lustre: server umount lustre-OST0000 complete [ 5799.706380] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5801.756292] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5826.233191] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5828.757224] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 5831.626452] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5832.405282] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5834.102903] Lustre: server umount lustre-OST0000 complete [ 5836.768541] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5836.776345] LustreError: Skipped 3 previous similar messages [ 5836.778779] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5836.791279] Lustre: Skipped 5 previous similar messages [ 5842.036909] Lustre: server umount lustre-MDT0000 complete [ 5843.838745] LustreError: 204091:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775221684 with bad export cookie 8449420419573531099 [ 5843.843252] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5843.846377] LustreError: 204091:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5844.065792] Lustre: server umount lustre-MDT0001 complete [ 5848.259422] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 09:08:09 (1775221689) [ 5851.132050] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5851.388943] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5851.391680] Lustre: Skipped 6 previous similar messages [ 5853.080434] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 5856.492653] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5858.561129] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 5860.772722] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5862.245559] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5864.979731] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5867.595889] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 5869.703385] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5871.658343] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5876.864615] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 5 sec [ 5878.500988] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5878.584845] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5881.365201] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5881.429934] Lustre: lustre-OST0001: new disk, initializing [ 5881.432771] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5883.795933] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 5887.307823] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5888.968327] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5891.082511] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5891.086735] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5891.124206] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5892.116688] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 5893.667968] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5893.769977] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5896.166370] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5896.168659] Lustre: Skipped 8 previous similar messages [ 5900.616530] Lustre: server umount lustre-OST0001 complete [ 5901.285450] LustreError: 206554:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5901.296478] LustreError: 206554:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 5903.420935] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5903.519712] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5905.121235] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5905.207708] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5929.317329] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5931.751154] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 5933.957316] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5934.567152] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 5934.575436] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 5934.575511] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5934.589349] Lustre: Skipped 1 previous similar message [ 5934.732660] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5937.663166] Lustre: server umount lustre-OST0001 complete [ 5956.248338] Lustre: server umount lustre-OST0000 complete [ 5964.009426] Lustre: server umount lustre-MDT0000 complete [ 5965.348110] LustreError: 204940:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775221806 with bad export cookie 8449420419573532030 [ 5965.352310] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5965.352700] LustreError: 204940:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5971.514314] Lustre: server umount lustre-MDT0001 complete [ 5980.441044] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 09:10:21 (1775221821) [ 5982.892401] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5982.998308] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5983.014266] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5984.548533] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 5987.610195] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5987.643054] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5987.653665] Lustre: 212959:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5989.117251] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 5991.139719] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5992.452617] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5994.826347] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5994.892613] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5996.921883] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 5998.739118] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6002.694371] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6004.676514] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 6006.487706] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6007.007082] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6008.400122] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6008.476160] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6009.826097] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6009.903921] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6011.095638] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8f1160bcc800.ost_server_uuid 50 [ 6011.616922] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8f1160bcc800.ost_server_uuid in FULL state after 0 sec [ 6011.985928] Lustre: Permanently deactivating lustre-OST0001 [ 6011.991848] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6012.839593] Lustre: Permanently reactivating lustre-OST0001 [ 6018.017429] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6018.017712] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6018.021370] LustreError: Skipped 8 previous similar messages [ 6018.026904] Lustre: Skipped 9 previous similar messages [ 6028.255187] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6028.308543] Lustre: server umount lustre-OST0001 complete [ 6044.127132] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6044.183167] Lustre: server umount lustre-OST0000 complete [ 6051.680408] Lustre: server umount lustre-MDT0000 complete [ 6053.064200] LustreError: 214592:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775221894 with bad export cookie 8449420419573533297 [ 6053.066682] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6053.070758] LustreError: 214592:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6053.203514] Lustre: server umount lustre-MDT0001 complete [ 6062.337253] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 09:11:43 (1775221903) [ 6065.142333] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6065.251259] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6065.267088] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6065.271648] Lustre: Skipped 5 previous similar messages [ 6066.774896] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 6070.059557] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6070.101965] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6070.103660] Lustre: Skipped 1 previous similar message [ 6070.114991] Lustre: 219062:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6071.872228] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 6074.121891] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6075.722182] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6079.373200] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6079.524580] Lustre: Permanently deactivating lustre-OST0000 [ 6079.532714] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 6082.626728] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 6084.962288] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6086.664857] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6088.072675] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6088.122372] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6088.126199] Lustre: Skipped 1 previous similar message [ 6090.891153] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 6093.053496] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6093.344869] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 6098.807242] Lustre: Permanently reactivating lustre-OST0000 [ 6098.817835] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 6098.821151] Lustre: Skipped 2 previous similar messages [ 6099.643840] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6099.648558] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6099.653549] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6099.673160] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 6099.679134] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6110.979485] Lustre: server umount lustre-OST0001 complete [ 6118.351932] Lustre: server umount lustre-OST0000 complete [ 6119.769290] Lustre: server umount lustre-MDT0000 complete [ 6121.072381] LustreError: 219052:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775221962 with bad export cookie 8449420419573534942 [ 6121.075939] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6121.077393] LustreError: 219052:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6121.203526] Lustre: server umount lustre-MDT0001 complete [ 6124.948621] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing unload_modules_local [ 6126.041975] Key type lgssc unregistered [ 6126.167473] LNet: 223440:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6126.170868] LNetError: 223440:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6126.183299] LNet: Removed LNI 192.168.202.141@tcp [ 6126.484529] Key type .llcrypt unregistered [ 6126.485712] Key type ._llcrypt unregistered [ 6136.093240] Key type ._llcrypt registered [ 6136.094384] Key type .llcrypt registered [ 6136.139709] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 6136.605464] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6136.619643] alg: No test for adler32 (adler32-zlib) [ 6137.510774] Lustre: Lustre: Build Version: 2.17.51_74_g8e7ba6b [ 6137.616148] LNet: Added LNI 192.168.202.141@tcp [8/256/0/180] [ 6139.215149] Key type lgssc registered [ 6139.731214] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6144.004077] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6144.012654] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6145.216341] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6146.757258] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 6150.547590] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6150.605185] Lustre: Found index 1 for lustre-MDT0001, updating log [ 6150.625560] Lustre: 224873:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6150.636916] Lustre: Permanently deactivating lustre-MDT0001 [ 6150.643633] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 6150.770665] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6152.634206] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 6155.008146] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6155.245755] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 6155.249188] LustreError: 224915:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 6155.257989] LustreError: 224915:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 6155.262861] LustreError: 224915:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 10, retries 1, failed: rc = -108 [ 6156.445992] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6159.285669] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6159.427728] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6160.485913] LustreError: 226499:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6160.491146] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 6160.494112] LustreError: 226499:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6161.779509] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 6163.952489] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6165.985166] LustreError: 226499:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6166.717039] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6166.844114] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6169.109349] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 6170.921700] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 6170.922879] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 6170.926421] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 6171.228883] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6172.929934] Lustre: Permanently reactivating lustre-MDT0001 [ 6172.938990] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 6172.941389] Lustre: Skipped 2 previous similar messages [ 6181.257500] Lustre: lustre-MDT0001: Client 4aeb8403-bf22-4ac2-b087-c00e77187745 (at 192.168.202.41@tcp) reconnecting [ 6183.548251] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6184.149094] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6186.047860] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 6186.137400] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 6187.594261] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 6187.676123] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 6188.567052] Lustre: Permanently deactivating lustre-MDT0001 [ 6188.573557] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 6188.576189] Lustre: Skipped 2 previous similar messages [ 6196.245522] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 6204.898165] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6204.905066] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6204.920057] Lustre: Skipped 3 previous similar messages [ 6204.942732] Lustre: Skipped 3 previous similar messages [ 6209.023498] Lustre: server umount lustre-MDT0000 complete [ 6210.021270] LustreError: 224883:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6210.059050] LustreError: 224883:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6212.861317] LustreError: 224865:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775222053 with bad export cookie 6288758644153951159 [ 6212.878255] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6213.437926] Lustre: server umount lustre-MDT0001 complete [ 6222.238977] Lustre: server umount lustre-OST0000 complete [ 6232.870746] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 09:14:32 (1775222072) [ 6241.690407] Lustre: server umount lustre-OST0001 complete [ 6248.119334] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_hostid [ 6253.741963] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 6275.868672] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6276.014130] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6276.019279] Lustre: Skipped 2 previous similar messages [ 6276.034678] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6276.091582] Lustre: lustre-MDT0000: new disk, initializing [ 6276.146505] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6276.162978] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6278.728769] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 6285.876864] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6285.933537] Lustre: 232621:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6285.959705] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6285.961932] Lustre: Skipped 1 previous similar message [ 6286.014450] Lustre: lustre-MDT0001: new disk, initializing [ 6286.068102] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6286.095685] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6286.102499] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6288.947755] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 6293.868356] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6296.151872] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6300.590732] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6300.777809] Lustre: lustre-OST0000: new disk, initializing [ 6300.781232] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6300.933082] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6300.938784] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6300.961523] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 6304.622429] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 6310.102114] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6311.653076] LustreError: 232630:0:(mdt_handler.c:3264:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 6317.953411] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6318.063221] Lustre: lustre-OST0001: new disk, initializing [ 6318.065880] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6318.150515] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6318.157380] Lustre: Skipped 1 previous similar message [ 6321.695081] LustreError: 232630:0:(mdt_handler.c:3264:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 6321.710218] LustreError: 232628:0:(mdt_handler.c:3264:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 6322.005243] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 6325.323187] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6325.330581] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6325.376197] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6328.015946] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6329.177657] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6331.767189] LustreError: 232628:0:(mdt_handler.c:3264:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 6335.457958] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6335.464670] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6335.482426] Lustre: Skipped 1 previous similar message [ 6335.493799] Lustre: Skipped 1 previous similar message [ 6339.224816] Lustre: server umount lustre-OST0001 complete [ 6340.581478] LustreError: 234377:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6340.596970] LustreError: 234377:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6345.703809] LustreError: 234346:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6345.718837] LustreError: 234346:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6355.938429] LustreError: 234345:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6355.944706] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6355.948714] LustreError: 234345:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6355.960288] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6355.966793] Lustre: Skipped 2 previous similar messages [ 6361.063471] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6366.181076] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6366.186876] Lustre: Skipped 2 previous similar messages [ 6367.199529] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6367.271717] Lustre: server umount lustre-OST0000 complete [ 6371.296458] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6371.302792] Lustre: Skipped 2 previous similar messages [ 6375.980623] Lustre: server umount lustre-MDT0000 complete [ 6376.416383] LustreError: 232633:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6376.424943] LustreError: 232633:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 6378.109787] LustreError: 232613:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775222219 with bad export cookie 6288758644153953966 [ 6378.110063] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6378.123355] LustreError: 232613:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6384.378898] Lustre: server umount lustre-MDT0001 complete [ 6389.432882] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing unload_modules_local [ 6390.943197] Key type lgssc unregistered [ 6391.116595] LNet: 237268:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6391.121913] LNetError: 237268:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6391.135041] LNet: Removed LNI 192.168.202.141@tcp [ 6391.548240] Key type .llcrypt unregistered [ 6391.549520] Key type ._llcrypt unregistered [ 6413.293592] Key type ._llcrypt registered [ 6413.295831] Key type .llcrypt registered [ 6413.361890] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 6413.872654] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6413.901346] alg: No test for adler32 (adler32-zlib) [ 6414.820664] Lustre: Lustre: Build Version: 2.17.51_74_g8e7ba6b [ 6414.968143] LNet: Added LNI 192.168.202.141@tcp [8/256/0/180] [ 6416.568089] Key type lgssc registered [ 6417.101041] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6420.531286] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6420.538455] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6421.645995] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6421.660683] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6421.743589] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6423.436038] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 6427.098781] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6427.151701] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6427.168522] Lustre: 239435:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6427.282554] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6429.016777] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 6431.474982] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6432.880425] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6435.547392] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6435.623833] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6435.724732] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6437.330483] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4 to 0x280000400:33) [ 6437.716213] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 6439.543735] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6483.424361] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6483.433113] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6483.455162] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6483.936366] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6483.953836] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6483.983591] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6486.257845] Lustre: server umount lustre-OST0000 complete [ 6490.925924] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 6507.662476] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6508.084778] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6517.252287] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 6523.413843] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6524.451614] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6524.464738] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6524.464939] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6565.350519] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6565.351494] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6565.356415] LustreError: Skipped 1 previous similar message [ 6565.373191] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6565.373578] Lustre: Skipped 1 previous similar message [ 6567.698674] Lustre: server umount lustre-OST0000 complete [ 6570.976990] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6570.984709] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6571.011240] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6571.015435] Lustre: Skipped 1 previous similar message [ 6575.584237] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6575.585414] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6585.311621] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6585.577550] Lustre: server umount lustre-MDT0000 complete [ 6588.676960] LustreError: 240159:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775222429 with bad export cookie 11754199398646648812 [ 6588.678667] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6588.684449] LustreError: 240159:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6589.017465] Lustre: server umount lustre-MDT0001 complete [ 6596.861452] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing unload_modules_local [ 6599.071379] Key type lgssc unregistered [ 6599.340643] LNet: 244384:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6599.347666] LNetError: 244384:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6600.422089] LNet: Removed LNI 192.168.202.141@tcp [ 6601.055074] Key type .llcrypt unregistered [ 6601.057227] Key type ._llcrypt unregistered [ 6619.546539] Key type ._llcrypt registered [ 6619.549280] Key type .llcrypt registered [ 6619.621657] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 6620.350844] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6620.379587] alg: No test for adler32 (adler32-zlib) [ 6621.342334] Lustre: Lustre: Build Version: 2.17.51_74_g8e7ba6b [ 6621.494036] LNet: Added LNI 192.168.202.141@tcp [8/256/0/180] [ 6623.143247] Key type lgssc registered [ 6624.026213] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6629.359083] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6629.379374] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6630.713520] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6633.705755] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 6640.097827] LustreError: 245765:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6640.116243] LustreError: 245765:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6640.274549] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6640.504442] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6643.775132] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 6647.704655] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6650.525143] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6655.369030] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6655.626342] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6659.624153] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 6664.075738] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6675.125057] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:42 to 0x280000400:65) [ 6695.910248] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6695.912392] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6695.923073] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6695.929526] Lustre: Skipped 1 previous similar message [ 6701.028992] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6701.033885] Lustre: Skipped 1 previous similar message [ 6702.043967] Lustre: server umount lustre-OST0000 complete [ 6706.144858] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6706.158431] Lustre: Skipped 1 previous similar message [ 6706.167359] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6706.171330] Lustre: Skipped 1 previous similar message [ 6711.015655] Lustre: server umount lustre-MDT0000 complete [ 6711.265098] LustreError: 247399:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6711.278683] LustreError: 247399:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6714.249604] LustreError: 246482:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775222555 with bad export cookie 12833751768758512389 [ 6714.250327] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6714.258443] LustreError: 246482:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6714.573289] Lustre: server umount lustre-MDT0001 complete [ 6723.124400] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing unload_modules_local [ 6725.274853] Key type lgssc unregistered [ 6725.560932] LNet: 249738:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6725.568555] LNetError: 249738:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6725.593515] LNet: Removed LNI 192.168.202.141@tcp [ 6726.202391] Key type .llcrypt unregistered [ 6726.205599] Key type ._llcrypt unregistered [ 6739.535203] Key type ._llcrypt registered [ 6739.537182] Key type .llcrypt registered [ 6739.599516] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 6740.476713] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6740.517156] alg: No test for adler32 (adler32-zlib) [ 6741.464916] Lustre: Lustre: Build Version: 2.17.51_74_g8e7ba6b [ 6741.624457] LNet: Added LNI 192.168.202.141@tcp [8/256/0/180] [ 6743.247173] Key type lgssc registered [ 6744.142906] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6750.041551] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6750.057592] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6751.407596] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6754.523869] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 6760.930397] LustreError: 250921:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6760.959393] LustreError: 250921:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6761.164965] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6761.526126] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6764.760572] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 6769.347910] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6772.308604] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6777.358606] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6777.713700] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6782.855079] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 6787.686991] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6796.140603] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 6807.521466] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6807.527497] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6807.535517] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6808.035960] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 6808.038726] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6808.047416] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6811.634974] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6811.639654] Lustre: Skipped 1 previous similar message [ 6813.327375] Lustre: server umount lustre-OST0000 complete [ 6816.736784] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6816.746364] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6821.857930] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6821.861556] Lustre: Skipped 2 previous similar messages [ 6822.748330] Lustre: server umount lustre-MDT0000 complete [ 6825.276985] LustreError: 250901:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775222666 with bad export cookie 18330280244170890323 [ 6825.281803] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6825.286299] LustreError: 250901:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6825.567436] Lustre: server umount lustre-MDT0001 complete [ 6832.276687] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing unload_modules_local [ 6834.161859] Key type lgssc unregistered [ 6834.338824] LNet: 254266:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6834.354530] LNetError: 254266:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6834.367496] LNet: Removed LNI 192.168.202.141@tcp [ 6834.957903] Key type .llcrypt unregistered [ 6834.959215] Key type ._llcrypt unregistered [ 6850.755972] Key type ._llcrypt registered [ 6850.758400] Key type .llcrypt registered [ 6850.837672] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 6851.560062] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6851.584218] alg: No test for adler32 (adler32-zlib) [ 6852.503160] Lustre: Lustre: Build Version: 2.17.51_74_g8e7ba6b [ 6852.633192] LNet: Added LNI 192.168.202.141@tcp [8/256/0/180] [ 6854.255389] Key type lgssc registered [ 6854.861471] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6859.174304] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6859.183626] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6860.506398] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6862.647582] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 6867.626588] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6867.877614] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6870.613442] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 6874.217652] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6876.869996] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6881.308440] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6881.587201] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6885.378236] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 6885.668344] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 6888.742405] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6908.389797] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6908.390896] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6908.406527] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6908.411945] Lustre: Skipped 1 previous similar message [ 6908.422452] Lustre: Skipped 1 previous similar message [ 6913.508061] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6913.513787] Lustre: Skipped 1 previous similar message [ 6913.695500] Lustre: server umount lustre-OST0000 complete [ 6918.628807] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6918.631832] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6918.642695] Lustre: Skipped 1 previous similar message [ 6918.647397] Lustre: Skipped 1 previous similar message [ 6922.489956] Lustre: server umount lustre-MDT0000 complete [ 6923.751634] LustreError: 255645:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6923.766108] LustreError: 255645:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6924.906352] LustreError: 255623:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775222766 with bad export cookie 8697568043399431242 [ 6924.907191] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6924.921570] LustreError: 255623:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6925.141255] Lustre: server umount lustre-MDT0001 complete [ 6931.374409] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing unload_modules_local [ 6933.208859] Key type lgssc unregistered [ 6933.446672] LNet: 259591:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6933.452110] LNetError: 259591:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6933.465385] LNet: Removed LNI 192.168.202.141@tcp [ 6934.124461] Key type .llcrypt unregistered [ 6934.125801] Key type ._llcrypt unregistered [ 6946.244944] Key type ._llcrypt registered [ 6946.246480] Key type .llcrypt registered [ 6946.325787] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 6947.210300] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6947.282063] alg: No test for adler32 (adler32-zlib) [ 6948.281804] Lustre: Lustre: Build Version: 2.17.51_74_g8e7ba6b [ 6948.526676] LNet: Added LNI 192.168.202.141@tcp [8/256/0/180] [ 6950.287203] Key type lgssc registered [ 6951.370600] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6957.287948] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6957.308677] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6958.624941] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6961.650563] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 6967.714898] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6967.997522] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6971.034634] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 6974.840827] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6977.645950] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6982.852737] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6983.130566] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6984.178709] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 6987.897173] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 6992.194505] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7003.619603] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7003.626187] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7003.634285] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7003.652367] Lustre: Skipped 1 previous similar message [ 7007.792550] Lustre: server umount lustre-OST0000 complete [ 7013.858543] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7013.859264] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7013.875094] Lustre: Skipped 1 previous similar message [ 7013.884366] Lustre: Skipped 1 previous similar message [ 7018.982147] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7018.989157] Lustre: Skipped 1 previous similar message [ 7023.075046] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7023.081703] Lustre: Skipped 1 previous similar message [ 7025.119811] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7025.220314] Lustre: server umount lustre-MDT0000 complete [ 7027.940801] LustreError: 260753:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775222869 with bad export cookie 2009548894107731524 [ 7027.948686] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7027.950239] LustreError: 260753:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7028.178686] Lustre: server umount lustre-MDT0001 complete [ 7035.576620] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing unload_modules_local [ 7037.623986] Key type lgssc unregistered [ 7037.842584] LNet: 264136:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7037.847623] LNetError: 264136:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7037.868223] LNet: Removed LNI 192.168.202.141@tcp [ 7038.518204] Key type .llcrypt unregistered [ 7038.520989] Key type ._llcrypt unregistered [ 7047.164277] Key type ._llcrypt registered [ 7047.165664] Key type .llcrypt registered [ 7047.264803] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 7059.796813] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_hostid [ 7069.189866] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 7069.757668] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7069.778573] alg: No test for adler32 (adler32-zlib) [ 7070.714948] Lustre: Lustre: Build Version: 2.17.51_74_g8e7ba6b [ 7070.872743] LNet: Added LNI 192.168.202.141@tcp [8/256/0/180] [ 7072.503232] Key type lgssc registered [ 7073.250349] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7094.591342] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7094.609143] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7095.798557] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7095.824267] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7095.882500] Lustre: lustre-MDT0000: new disk, initializing [ 7095.943415] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7095.959137] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7098.575362] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 7105.532655] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7105.609933] Lustre: 267614:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7105.646963] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7105.650773] Lustre: Skipped 1 previous similar message [ 7105.698218] Lustre: lustre-MDT0001: new disk, initializing [ 7105.768202] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7105.798211] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7105.810322] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7108.400450] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 7112.834432] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7115.124908] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7118.724083] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7118.850751] Lustre: lustre-OST0000: new disk, initializing [ 7118.853489] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7118.893946] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7122.395251] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 7123.982674] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7123.988535] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7124.052243] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7126.919685] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7129.075128] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7129.179942] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7131.106937] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7131.219980] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7134.178062] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7134.180052] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7134.182158] LustreError: Skipped 1 previous similar message [ 7134.183150] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7134.195537] Lustre: Skipped 1 previous similar message [ 7138.297853] Lustre: server umount lustre-OST0000 complete [ 7141.857400] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7141.863719] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7141.873492] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7141.876889] Lustre: Skipped 1 previous similar message [ 7144.416243] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7144.416728] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7144.428780] Lustre: Skipped 1 previous similar message [ 7146.742312] Lustre: server umount lustre-MDT0000 complete [ 7148.820328] LustreError: 267607:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775222989 with bad export cookie 11307600691378719689 [ 7148.828884] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7149.032542] Lustre: server umount lustre-MDT0001 complete [ 7153.816188] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 09:29:54 (1775222994) [ 7157.889467] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7158.205589] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7160.377467] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 7163.364649] LustreError: 271618:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7163.376046] LustreError: 271618:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7165.699702] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7165.906927] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7168.501394] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 7172.332684] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7175.027385] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7179.366031] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7179.596505] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7182.846558] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 7186.248281] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7199.201655] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7199.209634] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7199.219766] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7201.256370] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7202.620556] Lustre: server umount lustre-OST0000 complete [ 7205.160703] Lustre: server umount lustre-MDT0000 complete [ 7206.368381] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7206.370234] LustreError: 271618:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7206.373570] Lustre: Skipped 2 previous similar messages [ 7206.386563] LustreError: 271618:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7207.441680] LustreError: 273229:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775223048 with bad export cookie 11307600691378720571 [ 7207.444500] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7207.448523] LustreError: 273229:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 7207.638102] Lustre: server umount lustre-MDT0001 complete [ 7213.169936] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing unload_modules_local [ 7214.472859] Key type lgssc unregistered [ 7214.640033] LNet: 274792:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7214.648302] LNetError: 274792:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7214.663664] LNet: Removed LNI 192.168.202.141@tcp [ 7215.128275] Key type .llcrypt unregistered [ 7215.130822] Key type ._llcrypt unregistered [ 7229.809649] Key type ._llcrypt registered [ 7229.810918] Key type .llcrypt registered [ 7229.861727] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_hostid [ 7238.412594] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 7238.862338] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7238.870076] alg: No test for adler32 (adler32-zlib) [ 7239.774961] Lustre: Lustre: Build Version: 2.17.51_74_g8e7ba6b [ 7239.918634] LNet: Added LNI 192.168.202.141@tcp [8/256/0/180] [ 7241.511160] Key type lgssc registered [ 7242.062742] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7268.272426] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7268.292474] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7269.456476] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7269.476514] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7269.534369] Lustre: lustre-MDT0000: new disk, initializing [ 7269.603658] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7269.614476] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7272.268899] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 7278.893914] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7278.945764] Lustre: 278573:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7278.972494] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7278.975046] Lustre: Skipped 1 previous similar message [ 7279.020048] Lustre: lustre-MDT0001: new disk, initializing [ 7279.112293] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7279.144125] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7279.150258] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7284.967784] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 7297.579440] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7302.602487] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7310.388381] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7310.941953] Lustre: lustre-OST2710: new disk, initializing [ 7310.948882] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 7311.055918] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 7318.169225] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 7320.365982] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 7320.377273] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 7320.611734] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 7327.820448] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 7339.673630] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7340.163584] Lustre: lustre-OST03e8: new disk, initializing [ 7340.181858] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 7340.360735] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 7348.906578] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 7348.926621] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 7349.149850] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 7350.339738] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 7357.862544] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 7365.823667] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 7366.068607] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7371.212841] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 7371.448426] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7376.208132] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 7376.367530] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7380.919577] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 7381.197522] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7385.064154] Lustre: lustre-MDT0000-lwp-OST2710: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7385.068179] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7385.078929] Lustre: Skipped 3 previous similar messages [ 7385.096948] Lustre: Skipped 3 previous similar messages [ 7390.181534] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7390.189026] Lustre: Skipped 3 previous similar messages [ 7391.195474] Lustre: server umount lustre-MDT0000 complete [ 7394.619509] LustreError: 278564:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775223235 with bad export cookie 16928553867951729943 [ 7394.622056] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7394.631727] LustreError: 278564:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7394.929171] Lustre: server umount lustre-MDT0001 complete [ 7409.592386] Lustre: server umount lustre-OST2710 complete [ 7411.487887] Lustre: 276374:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775223236/real 1775223236] req@000000000adb7571 x1861456317454592/t0(0) o400->lustre-MDT0001-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1775223252 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7411.534202] Lustre: lustre-MDT0001-lwp-OST03e8: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7413.533402] Lustre: server umount lustre-OST03e8 complete [ 7431.405959] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_hostid [ 7442.503629] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 7482.158712] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7482.402778] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7482.434293] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7482.538320] Lustre: lustre-MDT0000: new disk, initializing [ 7482.596602] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7482.645127] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7488.041304] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 7498.458960] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7498.583338] Lustre: 286305:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7498.633848] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7498.637747] Lustre: Skipped 1 previous similar message [ 7498.711640] Lustre: lustre-MDT0001: new disk, initializing [ 7498.815189] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7498.862972] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7498.884819] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7503.330313] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 7510.328258] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7513.818717] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7520.232394] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7520.459139] Lustre: lustre-OST0000: new disk, initializing [ 7520.464704] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7520.546176] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7526.007878] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7526.027794] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7526.166567] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7526.970303] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 7534.438532] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7538.811521] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7539.072898] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7543.027110] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7543.275371] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7546.849047] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7546.857101] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7546.866351] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7551.222560] Lustre: server umount lustre-OST0000 complete [ 7557.089107] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7557.090478] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7557.099980] Lustre: Skipped 2 previous similar messages [ 7557.106887] Lustre: Skipped 2 previous similar messages [ 7561.485265] Lustre: server umount lustre-MDT0000 complete [ 7565.247352] LustreError: 286296:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775223406 with bad export cookie 16928553867951731623 [ 7565.250660] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7565.263468] LustreError: 286296:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7565.715406] Lustre: server umount lustre-MDT0001 complete [ 7574.206552] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 09:36:54 (1775223414) [ 7576.150769] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 7578.230275] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 09:36:57 (1775223417) [ 7581.959318] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 7607.447662] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7607.696686] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7607.728037] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7607.882759] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7612.215804] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 7620.871949] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7621.029060] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7621.083832] Lustre: 291734:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7625.047061] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 7630.530489] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7633.967983] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7654.701791] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7654.917580] Lustre: Denying initial registration attempt from nid 192.168.202.141@tcp, specified as failover [ 7654.931902] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 7654.939694] LustreError: 293346:0:(tgt_mount.c:2545:server_fill_super()) Unable to start targets: -99 [ 7654.947770] LustreError: 293346:0:(tgt_mount.c:2063:server_put_super()) no obd lustre-OST0000 [ 7654.953899] LustreError: 293346:0:(tgt_mount.c:114:server_deregister_mount()) lustre-OST0000 not registered [ 7655.046442] Lustre: server umount lustre-OST0000 complete [ 7655.048888] LustreError: 293346:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 7662.565827] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7662.581655] LustreError: Skipped 1 previous similar message [ 7662.585616] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7662.606851] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7668.030913] Lustre: server umount lustre-MDT0000 complete [ 7669.736838] LustreError: 291740:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7669.766103] LustreError: 291740:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 7671.552187] LustreError: 291726:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775223512 with bad export cookie 16928553867951732512 [ 7671.561977] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7671.565538] LustreError: 291726:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7671.915728] Lustre: server umount lustre-MDT0001 complete [ 7680.808778] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing unload_modules_local [ 7683.278937] Key type lgssc unregistered [ 7683.569849] LNet: 294449:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7683.576222] LNetError: 294449:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7683.601957] LNet: Removed LNI 192.168.202.141@tcp [ 7684.381900] Key type .llcrypt unregistered [ 7684.383642] Key type ._llcrypt unregistered [ 7694.744049] Key type ._llcrypt registered [ 7694.746171] Key type .llcrypt registered [ 7694.867863] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 7695.903818] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7695.913317] alg: No test for adler32 (adler32-zlib) [ 7697.010448] Lustre: Lustre: Build Version: 2.17.51_74_g8e7ba6b [ 7697.210548] LNet: Added LNI 192.168.202.141@tcp [8/256/0/180] [ 7698.848597] Key type lgssc registered [ 7699.818776] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7755.271145] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 7762.498880] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7762.529469] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7763.823916] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7763.860482] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7764.053350] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7767.578953] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 7774.475942] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7774.569138] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7774.602337] Lustre: 297188:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7774.814762] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7778.980768] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 7783.618338] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7786.537584] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7805.117921] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7805.371871] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7805.533232] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7810.890987] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 7815.955239] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7821.281320] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7821.288859] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7821.314184] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7821.798676] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7821.801153] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7821.825930] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7826.920148] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7826.929561] Lustre: Skipped 1 previous similar message [ 7832.031262] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7832.036155] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7832.049356] Lustre: Skipped 1 previous similar message [ 7832.372084] Lustre: server umount lustre-OST0000 complete [ 7836.130295] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7836.141170] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7836.167427] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7842.416482] Lustre: server umount lustre-MDT0000 complete [ 7845.908601] LustreError: 297178:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775223687 with bad export cookie 18031606116088162705 [ 7845.914715] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7845.934099] LustreError: 297178:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7846.242874] Lustre: server umount lustre-MDT0001 complete [ 7856.176189] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing unload_modules_local [ 7858.596627] Key type lgssc unregistered [ 7858.828709] LNet: 300406:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7858.841494] LNetError: 300406:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7859.877713] LNet: Removed LNI 192.168.202.141@tcp [ 7860.637201] Key type .llcrypt unregistered [ 7860.639293] Key type ._llcrypt unregistered [ 7883.490993] Key type ._llcrypt registered [ 7883.494242] Key type .llcrypt registered [ 7883.558726] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 7884.547655] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7884.658785] alg: No test for adler32 (adler32-zlib) [ 7885.931381] Lustre: Lustre: Build Version: 2.17.51_74_g8e7ba6b [ 7886.161449] LNet: Added LNI 192.168.202.141@tcp [8/256/0/180] [ 7887.815188] Key type lgssc registered [ 7888.976333] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7896.969209] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7896.992186] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7898.506240] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7901.809861] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 7908.845358] LustreError: 301807:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7908.865909] LustreError: 301807:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7909.587894] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7910.043454] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7914.633885] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 7919.233100] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7922.622765] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7930.993194] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7931.395929] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7938.003687] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 7943.816238] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7950.753378] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.41@tcp (stopping) [ 7950.840454] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7950.845553] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7953.890311] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7953.895620] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7953.902065] Lustre: Skipped 1 previous similar message [ 7953.921794] Lustre: Skipped 3 previous similar messages [ 7955.848247] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.41@tcp (stopping) [ 7956.888476] Lustre: server umount lustre-MDT0000 complete [ 7959.010614] LustreError: 301804:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7959.017365] LustreError: 301804:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7960.652771] LustreError: 301789:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775223801 with bad export cookie 9525283668870197928 [ 7960.657117] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7960.674545] LustreError: 301789:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7960.973483] LustreError: 303626:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.41@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7960.996825] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.41@tcp (stopping) [ 7964.131569] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7964.132731] LustreError: 303626:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7965.155042] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7965.170027] Lustre: Skipped 1 previous similar message [ 7967.108834] Lustre: server umount lustre-MDT0001 complete [ 7972.773189] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 7980.024602] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7985.631735] LustreError: 301116:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000f2497a1b x1861456994802688/t0(0) o250->MGC192.168.202.141@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7985.912774] LustreError: 303412:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7985.927966] LustreError: 303412:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7986.009724] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7986.063410] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 7990.236658] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 7995.236325] LustreError: 305066:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7995.242258] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7995.257053] LustreError: 305066:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 7998.299644] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7998.686981] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8002.879207] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 8004.087401] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 8008.228568] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8009.811083] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8012.990844] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8014.605924] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8019.429386] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8019.437109] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8019.450898] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8019.454339] Lustre: Skipped 1 previous similar message [ 8025.576985] Lustre: server umount lustre-OST0000 complete [ 8029.667020] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8029.693346] Lustre: Skipped 2 previous similar messages [ 8029.698140] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8034.788593] LustreError: 305067:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8034.798841] LustreError: 305067:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8034.891427] Lustre: server umount lustre-MDT0000 complete [ 8038.061216] LustreError: 303425:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775223879 with bad export cookie 9525283668870205628 [ 8038.066521] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8038.072799] LustreError: 303425:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8038.449943] Lustre: server umount lustre-MDT0001 complete [ 8048.180487] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing unload_modules_local [ 8051.248655] Key type lgssc unregistered [ 8051.589662] LNet: 307499:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8051.599895] LNetError: 307499:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8052.652738] LNet: Removed LNI 192.168.202.141@tcp [ 8053.408879] Key type .llcrypt unregistered [ 8053.411939] Key type ._llcrypt unregistered [ 8074.471966] Key type ._llcrypt registered [ 8074.474529] Key type .llcrypt registered [ 8074.582329] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 8075.911726] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8076.009506] alg: No test for adler32 (adler32-zlib) [ 8077.105914] Lustre: Lustre: Build Version: 2.17.51_74_g8e7ba6b [ 8077.373649] LNet: Added LNI 192.168.202.141@tcp [8/256/0/180] [ 8079.023286] Key type lgssc registered [ 8080.260156] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8087.732045] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8087.761841] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8089.244648] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8093.147673] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 8099.815089] LustreError: 308901:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8099.828412] LustreError: 308901:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8100.390582] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8100.734664] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8103.970692] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 8108.545446] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8111.607725] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8116.193179] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8116.202103] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8116.210777] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8119.265188] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8119.268807] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8119.284682] Lustre: Skipped 1 previous similar message [ 8120.495846] Lustre: server umount lustre-MDT0000 complete [ 8123.952502] LustreError: 308882:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775223965 with bad export cookie 18170995968063316395 [ 8123.962474] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8124.160290] Lustre: server umount lustre-MDT0001 complete [ 8134.869856] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8135.056802] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8135.081385] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8135.224841] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8138.489901] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 8146.000470] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8146.061991] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8146.081949] Lustre: 311257:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8146.277915] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8150.135539] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 8154.678306] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8158.035063] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8166.694773] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8166.936971] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 8166.947399] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 8166.951617] LustreError: 312915:0:(tgt_mount.c:2545:server_fill_super()) Unable to start targets: -2 [ 8166.955766] LustreError: 312915:0:(tgt_mount.c:2063:server_put_super()) no obd lustre-OST0000 [ 8166.967803] LustreError: 312915:0:(tgt_mount.c:114:server_deregister_mount()) lustre-OST0000 not registered [ 8167.028642] Lustre: server umount lustre-OST0000 complete [ 8167.031417] LustreError: 312915:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 8175.790677] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8176.000271] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8176.172951] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8181.228667] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 8183.459238] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 8185.766342] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8191.915250] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8191.980739] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 8192.035325] Lustre: lustre-OST0001: new disk, initializing [ 8192.040759] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8192.108045] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8196.836719] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 8197.699507] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8197.712337] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8197.790905] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8201.106489] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 8208.364686] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8208.364991] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8208.368781] LustreError: Skipped 1 previous similar message [ 8208.369987] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8208.376756] Lustre: Skipped 1 previous similar message [ 8208.393050] Lustre: Skipped 1 previous similar message [ 8209.651145] Lustre: server umount lustre-OST0001 complete [ 8213.480902] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8213.483773] LustreError: 313380:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8213.485396] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8213.493803] Lustre: Skipped 1 previous similar message [ 8213.516266] LustreError: 313380:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8218.595103] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8218.596503] LustreError: 313381:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8218.604065] Lustre: Skipped 2 previous similar messages [ 8218.618730] LustreError: 313381:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8218.791278] Lustre: server umount lustre-OST0000 complete [ 8223.203076] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8223.222226] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8227.753913] Lustre: server umount lustre-MDT0000 complete [ 8228.836267] LustreError: 311269:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8228.845851] LustreError: 311269:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8230.858289] LustreError: 311980:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775224071 with bad export cookie 18170995968063316948 [ 8230.865346] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8230.875311] LustreError: 311980:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 8231.166143] Lustre: server umount lustre-MDT0001 complete [ 8238.478800] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing unload_modules_local [ 8240.557863] Key type lgssc unregistered [ 8240.767207] LNet: 316165:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8240.771277] LNetError: 316165:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8240.779658] LNet: Removed LNI 192.168.202.141@tcp [ 8241.359295] Key type .llcrypt unregistered [ 8241.361201] Key type ._llcrypt unregistered [ 8296.852661] Key type ._llcrypt registered [ 8296.854209] Key type .llcrypt registered [ 8296.927951] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_hostid [ 8308.322500] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 8309.050990] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8309.146051] alg: No test for adler32 (adler32-zlib) [ 8310.182930] Lustre: Lustre: Build Version: 2.17.51_74_g8e7ba6b [ 8310.335422] LNet: Added LNI 192.168.202.141@tcp [8/256/0/180] [ 8311.967389] Key type lgssc registered [ 8312.763958] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8339.721800] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8339.742932] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8340.970546] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8340.985847] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8341.055288] Lustre: lustre-MDT0000: new disk, initializing [ 8341.111893] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8341.123850] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8344.834109] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 8353.570140] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8353.705229] Lustre: 321040:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8353.765925] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8353.770930] Lustre: Skipped 1 previous similar message [ 8353.869874] Lustre: lustre-MDT0001: new disk, initializing [ 8353.954947] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8353.988332] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8353.994948] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8358.250457] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 8364.580950] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8367.408641] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8372.583929] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8372.776378] Lustre: lustre-OST0000: new disk, initializing [ 8372.779731] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8372.834879] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8377.694025] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 8383.075745] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8383.089703] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8383.190102] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8383.763788] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8387.047392] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8387.205258] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8390.450913] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8390.690948] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8393.699908] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8393.705516] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8393.713292] Lustre: Skipped 1 previous similar message [ 8393.714842] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8393.721510] LustreError: Skipped 1 previous similar message [ 8398.515623] Lustre: server umount lustre-OST0000 complete [ 8403.936913] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8403.949143] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8403.954241] Lustre: Skipped 2 previous similar messages [ 8407.670489] Lustre: server umount lustre-MDT0000 complete [ 8410.295850] LustreError: 321820:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775224251 with bad export cookie 1581975147527155667 [ 8410.303193] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8410.307212] LustreError: 321820:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8410.553943] Lustre: server umount lustre-MDT0001 complete [ 8417.465156] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 09:50:57 (1775224257) [ 8422.942327] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 09:51:03 (1775224263) [ 8429.234565] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8429.635039] LustreError: 325388:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8429.646911] LustreError: 325388:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 8429.696334] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8433.102943] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 8435.172930] LustreError: 325389:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8439.277772] LustreError: 325388:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8439.445302] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8439.668220] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8442.776754] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 8447.179663] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8449.989548] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8455.592360] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8455.904927] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8460.821702] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 8465.128678] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8481.913090] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 8483.189955] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 8484.426405] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 8486.016718] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 8487.485578] Lustre: Failing over lustre-MDT0000 [ 8487.793304] Lustre: server umount lustre-MDT0000 complete [ 8489.964350] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8489.966792] LustreError: 325388:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8489.976050] Lustre: Skipped 3 previous similar messages [ 8490.001538] LustreError: 325388:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8495.073850] LustreError: 325383:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8495.095443] LustreError: 325383:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 8504.677388] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8504.769332] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8505.011536] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8505.046311] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 8507.781120] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 8508.230623] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 8510.443792] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 8510.469817] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 8510.517671] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 8514.161566] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8515.303520] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8518.257248] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 8519.587021] Lustre: Failing over lustre-MDT0000 [ 8519.762184] Lustre: server umount lustre-MDT0000 complete [ 8520.676773] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8520.681466] LustreError: 325385:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8520.683279] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8520.693198] Lustre: Skipped 2 previous similar messages [ 8520.705987] LustreError: 325385:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 8535.907292] Lustre: 319269:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775224361/real 1775224361] req@00000000fabe2828 x1861457439436544/t0(0) o400->MGC192.168.202.141@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1775224377 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8535.941694] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8536.704643] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8538.498522] LustreError: 326999:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.41@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8538.519666] LustreError: 326999:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 12 previous similar messages [ 8546.483715] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8546.536152] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 8547.248254] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 8549.300109] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 8551.924492] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8551.928733] Lustre: Skipped 2 previous similar messages [ 8551.954565] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 8551.994138] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 8554.524242] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8555.961097] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8562.151053] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8562.155480] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8562.162347] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8562.168744] Lustre: Skipped 1 previous similar message [ 8566.192422] Lustre: server umount lustre-MDT0000 complete [ 8569.185812] LustreError: 325370:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775224410 with bad export cookie 1581975147527158383 [ 8569.194592] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8569.204175] LustreError: 325370:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8569.438936] Lustre: server umount lustre-MDT0001 complete [ 8583.086268] Lustre: server umount lustre-OST0000 complete [ 8595.796277] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8596.288758] LustreError: 331189:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8596.310455] LustreError: 331189:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 10 previous similar messages [ 8599.868561] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 8605.748703] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8608.598810] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 8612.169768] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8614.582628] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8618.635084] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8618.900339] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8618.914330] Lustre: Skipped 2 previous similar messages [ 8622.000190] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 8622.509764] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 8625.921668] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8636.389740] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8636.396616] Lustre: Skipped 1 previous similar message [ 8636.398830] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8636.400721] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8636.408389] Lustre: Skipped 3 previous similar messages [ 8640.249302] Lustre: server umount lustre-OST0000 complete [ 8646.626041] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8646.629536] Lustre: Skipped 1 previous similar message [ 8648.778377] Lustre: server umount lustre-MDT0000 complete [ 8651.212688] LustreError: 331171:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775224492 with bad export cookie 1581975147527159125 [ 8651.215729] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8651.223690] LustreError: 331171:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8651.405315] Lustre: server umount lustre-MDT0001 complete [ 8658.790370] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing unload_modules_local [ 8660.784371] Key type lgssc unregistered [ 8661.054598] LNet: 334368:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8661.064974] LNetError: 334368:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8662.122106] LNet: Removed LNI 192.168.202.141@tcp [ 8662.841513] Key type .llcrypt unregistered [ 8662.843184] Key type ._llcrypt unregistered [ 8681.388313] Key type ._llcrypt registered [ 8681.391886] Key type .llcrypt registered [ 8681.464481] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 8682.229399] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8682.255786] alg: No test for adler32 (adler32-zlib) [ 8683.285509] Lustre: Lustre: Build Version: 2.17.51_74_g8e7ba6b [ 8683.462937] LNet: Added LNI 192.168.202.141@tcp [8/256/0/180] [ 8685.087254] Key type lgssc registered [ 8685.791109] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8691.399385] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8691.417043] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8692.897413] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8696.231731] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 8702.436722] LustreError: 335848:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8702.447671] LustreError: 335848:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8702.962330] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8703.194853] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8706.125542] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 8710.279749] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8712.999766] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8718.019136] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8718.212059] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8722.852702] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 8726.858797] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8737.730097] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 8748.003555] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8748.004235] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8748.013684] Lustre: Skipped 2 previous similar messages [ 8748.017912] Lustre: Skipped 1 previous similar message [ 8751.562253] Lustre: server umount lustre-MDT0000 complete [ 8753.120365] LustreError: 335843:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8753.127887] LustreError: 335843:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8753.748425] LustreError: 335829:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775224594 with bad export cookie 16082162439496383971 [ 8753.751267] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8753.754653] LustreError: 335829:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8753.958795] Lustre: server umount lustre-MDT0001 complete [ 8766.514714] Lustre: server umount lustre-OST0000 complete [ 8775.828459] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8776.117317] LustreError: 339240:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8776.174124] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8778.863531] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 8781.281823] LustreError: 339241:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8784.401552] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8784.647773] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8787.289824] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 8790.534534] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8792.859648] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8797.036934] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8797.254672] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8801.347674] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 8801.387089] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 8805.414260] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8815.584819] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8815.587531] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8815.602201] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8815.607049] Lustre: Skipped 1 previous similar message [ 8820.945830] Lustre: server umount lustre-MDT0000 complete [ 8821.219697] LustreError: 339240:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8821.231716] LustreError: 339240:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8823.197909] LustreError: 339221:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775224664 with bad export cookie 16082162439496385084 [ 8823.201344] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8823.207295] LustreError: 339221:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8823.474319] Lustre: server umount lustre-MDT0001 complete [ 8836.801325] Lustre: server umount lustre-OST0000 complete [ 8846.865769] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8847.128631] LustreError: 342814:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8847.173783] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8849.544798] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 8854.576712] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8857.980304] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 8861.753196] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8864.519658] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8868.902493] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8872.185888] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 8873.051251] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 8876.649333] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8882.656873] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8882.664148] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8882.679224] Lustre: Skipped 2 previous similar messages [ 8882.684224] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8882.688751] Lustre: Skipped 3 previous similar messages [ 8884.705959] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8884.708859] LustreError: lustre-OST0000-osc-MDT0001: operation ost_create to node 0@lo failed: rc = -107 [ 8884.709421] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8884.711404] Lustre: Skipped 1 previous similar message [ 8884.735187] LustreError: Skipped 1 previous similar message [ 8885.962914] Lustre: server umount lustre-OST0000 complete [ 8889.824232] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8889.833946] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8889.834768] Lustre: Skipped 1 previous similar message [ 8889.846944] Lustre: Skipped 1 previous similar message [ 8894.595915] Lustre: server umount lustre-MDT0000 complete [ 8894.944161] LustreError: 342814:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8894.954961] LustreError: 342814:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8897.019569] LustreError: 342797:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775224738 with bad export cookie 16082162439496386085 [ 8897.023649] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8897.027754] LustreError: 342797:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8897.038354] LustreError: 343526:0:(osp_precreate.c:651:osp_precreate_send()) lustre-OST0000-osc-MDT0001: can't precreate: rc = -5 [ 8897.047153] LustreError: 343526:0:(osp_precreate.c:1355:osp_precreate_thread()) lustre-OST0000-osc-MDT0001: cannot precreate objects: rc = -5 [ 8897.265115] Lustre: server umount lustre-MDT0001 complete [ 8904.021379] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing unload_modules_local [ 8906.066332] Key type lgssc unregistered [ 8906.257678] LNet: 345988:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8906.265052] LNetError: 345988:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8906.274803] LNet: Removed LNI 192.168.202.141@tcp [ 8906.758749] Key type .llcrypt unregistered [ 8906.759933] Key type ._llcrypt unregistered [ 8923.073103] Key type ._llcrypt registered [ 8923.077896] Key type .llcrypt registered [ 8923.161800] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 8923.770992] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8923.812088] alg: No test for adler32 (adler32-zlib) [ 8924.758862] Lustre: Lustre: Build Version: 2.17.51_74_g8e7ba6b [ 8924.924321] LNet: Added LNI 192.168.202.141@tcp [8/256/0/180] [ 8926.551261] Key type lgssc registered [ 8927.240476] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8931.771365] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8931.788504] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8931.801249] LustreError: 347402:0:(osd_handler.c:8532:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 8931.817975] LustreError: 347402:0:(obd_config.c:843:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 8931.831115] LustreError: 347402:0:(obd_mount.c:250:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 8931.840679] LustreError: 347402:0:(tgt_mount.c:2513:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 8931.850361] LustreError: 347402:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 8936.480597] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8936.489343] LustreError: 347799:0:(osd_handler.c:8532:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 8936.506119] LustreError: 347799:0:(obd_config.c:843:class_setup()) setup lustre-OST0000-osd failed (-22) [ 8936.509708] LustreError: 347799:0:(obd_mount.c:250:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 8936.513563] LustreError: 347799:0:(tgt_mount.c:2513:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 8936.517280] LustreError: 347799:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 8948.159861] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing unload_modules_local [ 8949.838972] Key type lgssc unregistered [ 8950.018908] LNet: 348783:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8950.025201] LNetError: 348783:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8950.042504] LNet: Removed LNI 192.168.202.141@tcp [ 8950.560548] Key type .llcrypt unregistered [ 8950.563341] Key type ._llcrypt unregistered [ 8961.890505] Key type ._llcrypt registered [ 8961.891704] Key type .llcrypt registered [ 8961.980020] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_hostid [ 8972.465569] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 8973.069499] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8973.101153] alg: No test for adler32 (adler32-zlib) [ 8974.059301] Lustre: Lustre: Build Version: 2.17.51_74_g8e7ba6b [ 8974.221501] LNet: Added LNI 192.168.202.141@tcp [8/256/0/180] [ 8975.863170] Key type lgssc registered [ 8976.603148] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8999.036829] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8999.057874] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9000.270969] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9000.296366] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9000.356961] Lustre: lustre-MDT0000: new disk, initializing [ 9000.422189] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9000.438572] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9003.670378] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 9011.671226] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9011.744149] Lustre: 351883:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9011.777498] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9011.779460] Lustre: Skipped 1 previous similar message [ 9011.830566] Lustre: lustre-MDT0001: new disk, initializing [ 9011.907949] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9011.943993] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9011.951191] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9015.400536] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 9021.210286] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9023.591090] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9028.095178] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9028.285898] Lustre: lustre-OST0000: new disk, initializing [ 9028.288116] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9028.336783] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9032.557832] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 9037.340275] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9037.351660] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9037.443744] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9038.263540] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9042.130521] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9042.398603] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9046.125529] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9046.561926] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9052.646410] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9052.655730] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9052.656133] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9055.740362] Lustre: server umount lustre-OST0000 complete [ 9062.884770] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9062.913501] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9062.926475] Lustre: Skipped 2 previous similar messages [ 9062.949895] Lustre: Skipped 2 previous similar messages [ 9068.003381] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9068.011233] Lustre: Skipped 1 previous similar message [ 9068.875447] Lustre: server umount lustre-MDT0000 complete [ 9074.949564] LustreError: 351874:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775224916 with bad export cookie 8541044105681249069 [ 9074.952829] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9074.958699] LustreError: 351874:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9075.543921] Lustre: server umount lustre-MDT0001 complete [ 9087.123272] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 9089.724476] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 10:02:08 (1775224928) [ 9097.632747] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9098.187648] LustreError: 355946:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9098.205128] LustreError: 355946:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9098.285406] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9103.330478] LustreError: 355947:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9103.965068] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 9108.467535] LustreError: 355946:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9113.577652] LustreError: 355947:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9113.799517] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9114.303530] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9118.834838] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 9124.793738] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9128.839957] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9135.804492] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9136.148894] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9142.214428] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 9147.456684] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9154.236273] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9154.418925] Lustre: lustre-OST0001: new disk, initializing [ 9154.422956] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9154.537085] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9160.665404] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 9163.883306] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9163.894586] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9163.970015] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9168.199302] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 9174.499555] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9174.501894] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9174.518355] LustreError: Skipped 1 previous similar message [ 9174.519824] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9174.533262] Lustre: Skipped 1 previous similar message [ 9177.732819] Lustre: server umount lustre-OST0001 complete [ 9179.618505] LustreError: 357541:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9179.628962] LustreError: 357541:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9189.857254] LustreError: 357541:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9189.884755] LustreError: 357541:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 9194.469793] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9194.475303] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9194.503232] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9194.509367] Lustre: Skipped 2 previous similar messages [ 9205.219623] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9205.229567] Lustre: Skipped 4 previous similar messages [ 9208.287160] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9208.441855] Lustre: server umount lustre-OST0000 complete [ 9215.464953] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9215.493427] Lustre: Skipped 2 previous similar messages [ 9218.861952] Lustre: server umount lustre-MDT0000 complete [ 9220.579879] LustreError: 358121:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9220.594356] LustreError: 358121:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 9222.724374] LustreError: 355928:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775225063 with bad export cookie 8541044105681249951 [ 9222.731607] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9222.733768] LustreError: 355928:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9229.192162] Lustre: server umount lustre-MDT0001 complete [ 9239.788083] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing unload_modules_local [ 9242.451482] Key type lgssc unregistered [ 9242.734842] LNet: 360303:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9242.744084] LNetError: 360303:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9242.774217] LNet: Removed LNI 192.168.202.141@tcp [ 9243.568105] Key type .llcrypt unregistered [ 9243.570173] Key type ._llcrypt unregistered [ 9293.406688] Key type ._llcrypt registered [ 9293.409457] Key type .llcrypt registered [ 9293.531383] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 9294.491583] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9294.613325] alg: No test for adler32 (adler32-zlib) [ 9295.722218] Lustre: Lustre: Build Version: 2.17.51_74_g8e7ba6b [ 9295.947767] LNet: Added LNI 192.168.202.141@tcp [8/256/0/180] [ 9297.579640] Key type lgssc registered [ 9298.634726] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9305.772830] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9305.788243] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9307.034897] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9307.049057] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9307.167198] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9311.414896] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 9320.170988] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9320.288067] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9320.354697] Lustre: 363085:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9320.638862] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9325.467692] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 9331.110633] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9335.217878] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9338.336700] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9338.337669] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9338.352699] Lustre: Skipped 1 previous similar message [ 9338.364993] Lustre: Skipped 1 previous similar message [ 9343.581952] Lustre: server umount lustre-MDT0000 complete [ 9347.434295] LustreError: 363077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775225188 with bad export cookie 14433902712401594034 [ 9347.450268] LustreError: 363077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9347.451609] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9347.820294] Lustre: server umount lustre-MDT0001 complete [ 9356.608808] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 10:06:36 (1775225196) [ 9362.856814] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9363.410114] LustreError: 365373:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9363.422855] LustreError: 365373:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9363.464692] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9368.150451] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 9368.557127] LustreError: 365374:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9373.669075] LustreError: 365373:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9377.535241] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9377.989841] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9383.040186] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 9389.065642] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9392.844403] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9399.751803] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9399.885564] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9400.086322] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9407.267467] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 9412.995964] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9420.633147] Lustre: Permanently deactivating lustre-OST0000 [ 9420.652383] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 9425.934351] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 9428.878399] LustreError: 367974:0:(mgs_llog.c:1942:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [ 9428.883988] LustreError: 367974:0:(mgs_llog.c:1997:mgs_replace_nids()) Only MGS is allowed to be started [ 9428.889871] LustreError: 367974:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 9432.400759] LustreError: 368022:0:(mgs_llog.c:1997:mgs_replace_nids()) Only MGS is allowed to be started [ 9432.408997] LustreError: 368022:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 9440.949702] Lustre: server umount lustre-OST0000 complete [ 9443.763614] LustreError: 368273:0:(mgs_llog.c:1997:mgs_replace_nids()) Only MGS is allowed to be started [ 9443.773458] LustreError: 368273:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 9449.954421] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9449.960681] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9449.987699] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9455.584266] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9455.600269] Lustre: Skipped 3 previous similar messages [ 9460.191171] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9460.705618] LustreError: 366986:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9460.718380] LustreError: 366986:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9460.766186] Lustre: server umount lustre-MDT0000 complete [ 9464.979253] LustreError: 365355:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775225306 with bad export cookie 14433902712401594615 [ 9464.980747] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9464.992411] LustreError: 365355:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9475.784768] Lustre: server umount lustre-MDT0001 complete [ 9485.781617] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9490.643502] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 9493.963288] LustreError: 369416:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [ 9501.025698] Lustre: 369755:0:(mgs_llog.c:1709:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.141@tcp [ 9503.398853] Lustre: server umount MGS complete [ 9516.299557] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9516.704411] LustreError: 370478:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9516.735808] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 9516.744706] Lustre: Skipped 1 previous similar message [ 9516.824891] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9521.747546] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 9527.269333] LustreError: 370479:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9527.291938] LustreError: 370479:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9530.397823] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9530.876877] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 9535.882825] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 9542.481643] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9546.462614] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9548.515298] Lustre: Permanently reactivating lustre-OST0000 [ 9548.524706] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 9548.529465] Lustre: Skipped 2 previous similar messages [ 9574.373162] LustreError: 371983:0:(osp_dev.c:694:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [ 9582.220255] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9582.626488] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9582.634224] Lustre: Skipped 1 previous similar message [ 9588.672669] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 9594.300344] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9599.968986] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9599.982594] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9600.001764] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9600.003585] Lustre: Skipped 2 previous similar messages [ 9605.111348] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9605.116674] Lustre: Skipped 2 previous similar messages [ 9605.651922] Lustre: server umount lustre-OST0000 complete [ 9610.212227] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9610.243659] Lustre: Skipped 1 previous similar message [ 9615.328844] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9615.334238] Lustre: Skipped 3 previous similar messages [ 9615.844109] Lustre: server umount lustre-MDT0000 complete [ 9619.072606] LustreError: 370459:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775225460 with bad export cookie 14433902712401595665 [ 9619.077199] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9619.082061] LustreError: 370459:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9619.515047] Lustre: server umount lustre-MDT0001 complete [ 9627.650844] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing unload_modules_local [ 9629.640272] Key type lgssc unregistered [ 9629.895669] LNet: 374487:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9629.900066] LNetError: 374487:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9629.912045] LNet: Removed LNI 192.168.202.141@tcp [ 9630.466377] Key type .llcrypt unregistered [ 9630.468374] Key type ._llcrypt unregistered [ 9643.586590] Key type ._llcrypt registered [ 9643.589586] Key type .llcrypt registered [ 9643.683903] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_hostid [ 9654.652944] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 9655.517867] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9655.530052] alg: No test for adler32 (adler32-zlib) [ 9656.640676] Lustre: Lustre: Build Version: 2.17.51_74_g8e7ba6b [ 9656.887557] LNet: Added LNI 192.168.202.141@tcp [8/256/0/180] [ 9658.551116] Key type lgssc registered [ 9659.308177] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9681.818327] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9681.860510] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9683.011799] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9683.031708] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9683.096187] Lustre: lustre-MDT0000: new disk, initializing [ 9683.165252] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9683.177647] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9686.100634] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 9692.992376] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9693.081190] Lustre: 377586:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9693.124032] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9693.127384] Lustre: Skipped 1 previous similar message [ 9693.176261] Lustre: lustre-MDT0001: new disk, initializing [ 9693.263696] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9693.286505] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9693.294162] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9695.604239] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 9700.366610] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9702.721686] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9706.699516] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9706.903726] Lustre: lustre-OST0000: new disk, initializing [ 9706.907860] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9706.971542] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9707.468778] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9707.476550] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9707.571062] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9710.688921] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 9715.762087] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9718.226965] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9718.354542] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9721.360759] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9721.527649] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9727.971423] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9727.977914] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9727.979517] LustreError: Skipped 1 previous similar message [ 9727.989894] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9727.993305] Lustre: Skipped 1 previous similar message [ 9733.097505] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9733.101442] Lustre: Skipped 1 previous similar message [ 9737.183146] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9737.289707] Lustre: server umount lustre-OST0000 complete [ 9743.328264] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9743.330938] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9743.336483] Lustre: Skipped 2 previous similar messages [ 9743.340896] Lustre: Skipped 1 previous similar message [ 9747.839921] Lustre: server umount lustre-MDT0000 complete [ 9750.019378] LustreError: 377579:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775225591 with bad export cookie 4722527185631511259 [ 9750.030041] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9750.215705] Lustre: server umount lustre-MDT0001 complete [ 9755.686834] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [ 9756.761927] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 10:13:17 (1775225597) [ 9761.394496] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9761.716676] LustreError: 381636:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9761.729462] LustreError: 381636:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9761.759051] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9764.300318] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 9766.890074] LustreError: 381637:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9769.340435] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9769.560419] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9771.709827] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 9775.048912] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9777.304159] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9781.268765] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9781.487987] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9785.126381] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 9788.763563] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9789.895452] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [ 9791.843546] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [ 9800.164840] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9800.168489] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9800.180358] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9800.183179] Lustre: Skipped 1 previous similar message [ 9801.798566] Lustre: server umount lustre-OST0000 complete [ 9805.284647] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9805.286180] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9805.287401] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9805.297606] Lustre: Skipped 2 previous similar messages [ 9810.319672] Lustre: server umount lustre-MDT0000 complete [ 9810.400365] LustreError: 381633:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9810.408090] LustreError: 381633:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9812.525500] LustreError: 381618:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775225653 with bad export cookie 4722527185631512141 [ 9812.532324] LustreError: 381618:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9812.534022] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9812.731764] Lustre: server umount lustre-MDT0001 complete [ 9820.237943] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing unload_modules_local [ 9821.812848] Key type lgssc unregistered [ 9822.001249] LNet: 385006:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9822.008729] LNetError: 385006:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9822.028824] LNet: Removed LNI 192.168.202.141@tcp [ 9822.612164] Key type .llcrypt unregistered [ 9822.613616] Key type ._llcrypt unregistered [ 9838.381538] Key type ._llcrypt registered [ 9838.384635] Key type .llcrypt registered [ 9838.462640] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing unload_modules_local [ 9838.644273] Key type .llcrypt unregistered [ 9838.645845] Key type ._llcrypt unregistered [ 9849.670897] Key type ._llcrypt registered [ 9849.672557] Key type .llcrypt registered [ 9849.723149] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 9850.228484] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9850.285788] alg: No test for adler32 (adler32-zlib) [ 9851.230438] Lustre: Lustre: Build Version: 2.17.51_74_g8e7ba6b [ 9851.359659] LNet: Added LNI 192.168.202.141@tcp [8/256/0/180] [ 9853.007173] Key type lgssc registered [ 9853.628727] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9858.215444] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9858.231275] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9859.523794] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9861.936670] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 9866.649165] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9866.821656] LustreError: 387160:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9866.837937] LustreError: 387160:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9866.841774] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9869.921500] LustreError: 387161:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9869.932828] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 9869.939061] LustreError: 387161:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9870.691778] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 9873.981590] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9875.429926] LustreError: 387160:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9875.451851] LustreError: 387160:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9878.275224] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9878.476886] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9881.121911] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 9890.784812] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9890.796115] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9890.802779] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9893.864031] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9893.865707] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9893.866778] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9896.439392] Lustre: server umount lustre-OST0000 complete [ 9899.022950] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 9899.032611] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9899.041272] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9899.044282] Lustre: Skipped 1 previous similar message [ 9904.096574] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9904.098463] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9904.110715] Lustre: Skipped 1 previous similar message [ 9909.217681] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9909.221371] Lustre: Skipped 1 previous similar message [ 9913.311166] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9913.467889] Lustre: server umount lustre-MDT0000 complete [ 9914.337502] LustreError: 389119:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9914.347540] LustreError: 389119:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9915.633623] LustreError: 387142:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775225756 with bad export cookie 15494963374011415976 [ 9915.637529] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9915.638549] LustreError: 387142:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9915.850730] Lustre: server umount lustre-MDT0001 complete [ 9922.328369] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing unload_modules_local [ 9924.268510] Key type lgssc unregistered [ 9924.469204] LNet: 390149:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9924.480175] LNetError: 390149:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9924.498893] LNet: Removed LNI 192.168.202.141@tcp [ 9924.996741] Key type .llcrypt unregistered [ 9924.998391] Key type ._llcrypt unregistered [ 9939.740764] Key type ._llcrypt registered [ 9939.742710] Key type .llcrypt registered [ 9939.803059] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 9940.625921] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9940.698610] alg: No test for adler32 (adler32-zlib) [ 9941.678552] Lustre: Lustre: Build Version: 2.17.51_74_g8e7ba6b [ 9941.863453] LNet: Added LNI 192.168.202.141@tcp [8/256/0/180] [ 9943.480361] Key type lgssc registered [ 9944.057347] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9949.069320] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9949.080642] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9964.576662] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 9969.695485] LustreError: 391500:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.141@tcp: failed processing log, type 4: rc = -110 [ 9996.319238] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9999.560284] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [10002.602025] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10006.367335] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10006.677640] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10006.718173] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [10008.974558] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [10011.817197] LustreError: 391527:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10011.825418] LustreError: 391527:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10013.849273] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10014.083208] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10016.213185] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [10019.462615] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10021.700578] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10027.488419] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10027.495392] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10027.513915] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10029.538886] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10029.539563] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10029.540656] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10032.604029] Lustre: server umount lustre-OST0000 complete [10036.195316] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [10036.203384] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10036.213625] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10036.216737] Lustre: Skipped 1 previous similar message [10037.261625] Lustre: server umount lustre-MDT0000 complete [10039.559417] LustreError: 391508:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775225880 with bad export cookie 5940424617065529210 [10039.567791] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10039.570067] LustreError: 391508:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10039.776610] Lustre: server umount lustre-MDT0001 complete [10045.401425] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing unload_modules_local [10046.857400] Key type lgssc unregistered [10047.086619] LNet: 394711:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10047.089915] LNetError: 394711:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10048.101735] LNet: Removed LNI 192.168.202.141@tcp [10048.574256] Key type .llcrypt unregistered [10048.575799] Key type ._llcrypt unregistered [10061.745263] Key type ._llcrypt registered [10061.746930] Key type .llcrypt registered [10061.817796] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [10062.332169] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10062.383054] alg: No test for adler32 (adler32-zlib) [10063.300779] Lustre: Lustre: Build Version: 2.17.51_74_g8e7ba6b [10063.426921] LNet: Added LNI 192.168.202.141@tcp [8/256/0/180] [10065.047161] Key type lgssc registered [10065.674876] Lustre: Echo OBD driver; http://www.lustre.org/ [10070.261631] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10070.275623] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10071.555059] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10073.679800] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [10078.324708] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10078.566328] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10080.443334] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [10083.105759] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10084.788540] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10088.156395] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10088.319686] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10090.896227] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [10093.401424] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10096.556769] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [10101.729640] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10101.732162] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10101.741929] Lustre: Skipped 1 previous similar message [10105.236179] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.41@tcp (stopping) [10105.238703] Lustre: Skipped 2 previous similar messages [10106.851899] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10106.856621] Lustre: Skipped 2 previous similar messages [10107.319906] Lustre: server umount lustre-MDT0000 complete [10108.424856] LustreError: 396807:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.41@tcp arrived at 1775225949 with bad export cookie 16254098385325649565 [10108.432530] LustreError: 396807:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10109.384177] LustreError: 396069:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775225950 with bad export cookie 16254098385325648928 [10109.387689] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10109.390402] LustreError: 396069:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10109.923705] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10109.929657] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10109.943255] Lustre: Skipped 1 previous similar message [10109.947528] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10111.969336] LustreError: 396090:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10111.978697] LustreError: 396090:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10114.976440] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10114.976972] LustreError: 398236:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10114.980121] Lustre: Skipped 1 previous similar message [10114.992974] LustreError: 398236:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10115.481280] Lustre: server umount lustre-OST0000 complete [10118.706764] Lustre: server umount lustre-MDT0001 complete [10123.954546] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing unload_modules_local [10125.371600] Key type lgssc unregistered [10125.522901] LNet: 399418:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10125.527808] LNetError: 399418:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10125.546981] LNet: Removed LNI 192.168.202.141@tcp [10126.006959] Key type .llcrypt unregistered [10126.008728] Key type ._llcrypt unregistered [10139.389287] Key type ._llcrypt registered [10139.391233] Key type .llcrypt registered [10139.454915] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [10139.836301] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10139.867789] alg: No test for adler32 (adler32-zlib) [10140.749508] Lustre: Lustre: Build Version: 2.17.51_74_g8e7ba6b [10140.847446] LNet: Added LNI 192.168.202.141@tcp [8/256/0/180] [10142.439188] Key type lgssc registered [10142.954459] Lustre: Echo OBD driver; http://www.lustre.org/ [10146.781946] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10146.795564] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10148.038171] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10150.250321] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [10154.524773] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10154.720086] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10156.884539] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [10159.858577] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10161.790967] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10165.207450] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10165.414842] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10168.450495] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [10171.269694] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10173.611909] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [10179.560624] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [10179.565333] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10179.578570] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10183.043742] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.41@tcp (stopping) [10184.164514] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10184.165750] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10185.427183] Lustre: server umount lustre-MDT0001 complete [10189.281954] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10189.285772] LustreError: 401517:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10189.302248] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10189.302257] Lustre: Skipped 1 previous similar message [10189.315831] LustreError: 401517:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10194.402829] LustreError: 400794:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10194.403170] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10194.630607] Lustre: server umount lustre-OST0000 complete [10196.913678] Lustre: server umount lustre-MDT0000 complete [10204.333885] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing unload_modules_local [10205.832237] Key type lgssc unregistered [10206.006783] LNet: 404122:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10206.011791] LNetError: 404122:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10206.026162] LNet: Removed LNI 192.168.202.141@tcp [10206.518164] Key type .llcrypt unregistered [10206.520100] Key type ._llcrypt unregistered [10219.419612] Key type ._llcrypt registered [10219.420751] Key type .llcrypt registered [10219.489408] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_hostid [10228.568423] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [10229.171295] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10229.239386] alg: No test for adler32 (adler32-zlib) [10230.159111] Lustre: Lustre: Build Version: 2.17.51_74_g8e7ba6b [10230.276101] LNet: Added LNI 192.168.202.141@tcp [8/256/0/180] [10231.895174] Key type lgssc registered [10232.599707] Lustre: Echo OBD driver; http://www.lustre.org/ [10254.562490] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [10263.124660] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10263.167922] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10264.317264] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10264.348920] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10264.393319] Lustre: lustre-MDT0000: new disk, initializing [10264.441717] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10264.455807] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10266.624164] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [10275.155672] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10275.344390] Lustre: lustre-OST0000: new disk, initializing [10275.347164] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10275.349855] Lustre: Skipped 1 previous similar message [10275.400169] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10278.955721] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [10279.999952] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [10280.008668] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [10280.025476] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [10283.255623] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10290.988934] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10291.054476] Lustre: 408062:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10291.105115] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10291.154810] Lustre: lustre-MDT0001: new disk, initializing [10291.232995] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10291.257071] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [10291.263820] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [10293.514337] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [10298.463551] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [10298.592876] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [10300.581111] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [10300.707383] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [10306.960621] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.41@tcp (stopping) [10307.553912] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10307.556418] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10307.561146] Lustre: Skipped 1 previous similar message [10309.490297] Lustre: server umount lustre-MDT0001 complete [10317.793616] LustreError: 408069:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10317.802878] LustreError: 408069:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [10322.914906] LustreError: 410692:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10322.930534] LustreError: 410692:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10325.992253] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10325.999089] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10326.011468] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10326.016833] Lustre: Skipped 1 previous similar message [10328.033880] LustreError: 408074:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10328.034778] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10329.397922] Lustre: server umount lustre-OST0000 complete [10337.646090] Lustre: server umount lustre-MDT0000 complete [10345.346275] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing unload_modules_local [10347.069668] Key type lgssc unregistered [10347.285707] LNet: 412623:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10347.292235] LNetError: 412623:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10347.305499] LNet: Removed LNI 192.168.202.141@tcp [10347.821224] Key type .llcrypt unregistered [10347.827077] Key type ._llcrypt unregistered [10368.309244] Key type ._llcrypt registered [10368.310533] Key type .llcrypt registered [10368.358539] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing unload_modules_local [10368.522407] Key type .llcrypt unregistered [10368.523965] Key type ._llcrypt unregistered [10377.652071] Key type ._llcrypt registered [10377.653213] Key type .llcrypt registered [10377.712270] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [10378.202495] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10378.231081] alg: No test for adler32 (adler32-zlib) [10379.180866] Lustre: Lustre: Build Version: 2.17.51_74_g8e7ba6b [10379.314947] LNet: Added LNI 192.168.202.141@tcp [8/256/0/180] [10380.927155] Key type lgssc registered [10381.712792] Lustre: Echo OBD driver; http://www.lustre.org/ [10396.950367] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10396.965365] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10398.103816] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10398.123877] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10398.172573] Lustre: lustre-MDT0000: new disk, initializing [10398.219621] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10398.234433] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10400.448100] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [10406.519066] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10406.583161] Lustre: 415941:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10406.614940] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10406.616900] Lustre: Skipped 1 previous similar message [10406.656536] Lustre: lustre-MDT0001: new disk, initializing [10406.711223] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10406.752887] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10406.759578] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10408.730527] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [10413.016787] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10414.697855] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10418.140512] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10418.336822] Lustre: lustre-OST0000: new disk, initializing [10418.339961] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10418.379635] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10419.748711] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10419.758170] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10419.841740] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10421.712518] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [10426.501097] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10432.480371] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10432.483786] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10432.493926] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10435.555764] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10435.557035] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10435.561380] Lustre: Skipped 1 previous similar message [10435.566505] Lustre: Skipped 1 previous similar message [10436.942852] Lustre: server umount lustre-MDT0000 complete [10439.370629] LustreError: 415933:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775226280 with bad export cookie 13551414605121729765 [10439.375747] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10439.376091] LustreError: 415933:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10439.596569] Lustre: server umount lustre-MDT0001 complete [10451.983869] Lustre: server umount lustre-OST0000 complete [10457.048173] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 10:24:57 (1775226297) [10460.650858] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10460.906496] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10462.892700] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [10466.273087] LustreError: 419594:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10466.281689] LustreError: 419594:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10467.157030] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10467.331870] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10469.384745] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [10472.079909] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10473.725130] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10490.342279] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10490.468891] Lustre: Found index 0 for lustre-OST0000, updating log [10490.569622] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10493.571931] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [10496.386762] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10504.936468] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [10510.307333] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10510.307671] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10510.312155] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10510.322199] Lustre: Skipped 1 previous similar message [10511.487785] Lustre: server umount lustre-OST0000 complete [10527.712390] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10527.717412] Lustre: Skipped 1 previous similar message [10527.717585] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10527.721486] Lustre: Skipped 3 previous similar messages [10531.300933] Lustre: server umount lustre-MDT0000 complete [10532.833897] LustreError: 420750:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10532.842103] LustreError: 420750:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10533.153820] LustreError: 421205:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775226374 with bad export cookie 13551414605121731158 [10533.156432] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10533.158496] LustreError: 421205:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10533.337132] Lustre: server umount lustre-MDT0001 complete [10537.158940] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 10:26:17 (1775226377) [10553.375299] LNet: 1 local NIs in recovery (showing 1): 192.168.202.141@tcp [11126.778837] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11126.995062] LustreError: 425677:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11127.034470] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11128.856937] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [11132.389484] LustreError: 425677:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11132.399800] LustreError: 425677:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11133.262519] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11133.435705] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11135.316113] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [11138.110791] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11139.852079] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11156.075272] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11156.188173] Lustre: Found index 0 for lustre-OST0000, updating log [11156.328480] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11158.884705] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [11161.102605] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11165.664133] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11165.667526] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11165.675836] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11168.027442] Lustre: server umount lustre-OST0000 complete [11170.464477] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11170.466882] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11170.469784] Lustre: Skipped 1 previous similar message [11171.488227] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11171.491073] Lustre: Skipped 2 previous similar messages [11173.536175] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11173.538274] Lustre: Skipped 3 previous similar messages [11175.583982] LustreError: 425678:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11175.591439] LustreError: 425678:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11175.605945] Lustre: server umount lustre-MDT0000 complete [11177.268284] LustreError: 425657:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775227018 with bad export cookie 13551414605121732243 [11177.271126] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11177.274746] LustreError: 425657:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11177.411318] Lustre: server umount lustre-MDT0001 complete [11179.039236] LNet: 1 local NIs in recovery (showing 1): 192.168.202.141@tcp [11196.196250] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_hostid [11199.983392] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [11215.460829] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 10:37:36 (1775227056) [11222.468941] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing unload_modules_local [11223.696685] Key type lgssc unregistered [11223.865929] LNet: 431903:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11223.868848] LNetError: 431903:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11286.373434] LNet: Removed LNI 192.168.202.141@tcp [11286.747354] Key type .llcrypt unregistered [11286.748738] Key type ._llcrypt unregistered [11294.471448] Key type ._llcrypt registered [11294.473815] Key type .llcrypt registered [11294.529236] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [11294.996733] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11295.013074] alg: No test for adler32 (adler32-zlib) [11295.887197] Lustre: Lustre: Build Version: 2.17.51_74_g8e7ba6b [11295.992674] LNet: Added LNI 192.168.202.141@tcp [8/256/0/180] [11297.591172] Key type lgssc registered [11298.094690] Lustre: Echo OBD driver; http://www.lustre.org/ [11299.860573] LNet: Added LNI 192.168.202.141@tcp100 [8/256/0/180] [11300.195800] LNet: Added LNI 192.168.202.141@tcp101 [8/256/0/180] [11300.588542] LNet: Added LNI 192.168.202.141@tcp102 [8/256/0/180] [11301.004968] LNet: Added LNI 192.168.202.141@tcp103 [8/256/0/180] [11301.447427] LNet: Added LNI 192.168.202.141@tcp104 [8/256/0/180] [11301.892527] LNet: Added LNI 192.168.202.141@tcp105 [8/256/0/180] [11302.315776] LNet: Added LNI 192.168.202.141@tcp106 [8/256/0/180] [11302.716566] LNet: Added LNI 192.168.202.141@tcp107 [8/256/0/180] [11303.105703] LNet: Added LNI 192.168.202.141@tcp108 [8/256/0/180] [11303.490742] LNet: Added LNI 192.168.202.141@tcp109 [8/256/0/180] [11303.904348] LNet: Added LNI 192.168.202.141@tcp110 [8/256/0/180] [11304.309523] LNet: Added LNI 192.168.202.141@tcp111 [8/256/0/180] [11304.694502] LNet: Added LNI 192.168.202.141@tcp112 [8/256/0/180] [11305.039140] LNet: Added LNI 192.168.202.141@tcp113 [8/256/0/180] [11305.384213] LNet: Added LNI 192.168.202.141@tcp114 [8/256/0/180] [11305.732921] LNet: Added LNI 192.168.202.141@tcp115 [8/256/0/180] [11306.089183] LNet: Added LNI 192.168.202.141@tcp116 [8/256/0/180] [11306.412570] LNet: Added LNI 192.168.202.141@tcp117 [8/256/0/180] [11306.764752] LNet: Added LNI 192.168.202.141@tcp118 [8/256/0/180] [11307.093369] LNet: Added LNI 192.168.202.141@tcp119 [8/256/0/180] [11307.433791] LNet: Added LNI 192.168.202.141@tcp120 [8/256/0/180] [11307.806645] LNet: Added LNI 192.168.202.141@tcp121 [8/256/0/180] [11308.206320] LNet: Added LNI 192.168.202.141@tcp122 [8/256/0/180] [11308.571206] LNet: Added LNI 192.168.202.141@tcp123 [8/256/0/180] [11308.986947] LNet: Added LNI 192.168.202.141@tcp124 [8/256/0/180] [11309.440649] LNet: Added LNI 192.168.202.141@tcp125 [8/256/0/180] [11309.848453] LNet: Added LNI 192.168.202.141@tcp126 [8/256/0/180] [11310.330975] LNet: Added LNI 192.168.202.141@tcp127 [8/256/0/180] [11310.787794] LNet: Added LNI 192.168.202.141@tcp128 [8/256/0/180] [11311.183323] LNet: Added LNI 192.168.202.141@tcp129 [8/256/0/180] [11311.572945] LNet: Added LNI 192.168.202.141@tcp130 [8/256/0/180] [11311.962188] LNet: Added LNI 192.168.202.141@tcp131 [8/256/0/180] [11312.303608] LNet: Added LNI 192.168.202.141@tcp132 [8/256/0/180] [11312.658464] LNet: Added LNI 192.168.202.141@tcp133 [8/256/0/180] [11313.011259] LNet: Added LNI 192.168.202.141@tcp134 [8/256/0/180] [11313.354190] LNet: Added LNI 192.168.202.141@tcp135 [8/256/0/180] [11315.977412] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11315.985670] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11317.032469] Lustre: 434884:0:(lustre_peer.c:105:class_add_uuid()) 192.168.202.141@tcp: can't add NID 192.168.202.141@tcp130: rc = -75 [11317.109267] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11317.123202] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11317.166269] Lustre: lustre-MDT0000: new disk, initializing [11317.207139] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11317.215824] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11318.927847] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [11323.869965] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11323.951318] Lustre: 434900:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11323.992069] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11323.996277] Lustre: Skipped 1 previous similar message [11324.049723] Lustre: lustre-MDT0001: new disk, initializing [11324.073151] Lustre: 435684:0:(lustre_peer.c:105:class_add_uuid()) 192.168.202.141@tcp: can't add NID 192.168.202.141@tcp130: rc = -75 [11324.076979] Lustre: 435684:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [11324.140845] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11324.171762] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11324.180717] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11326.036312] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [11329.538052] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11330.874471] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11333.784963] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11333.918515] Lustre: lustre-OST0000: new disk, initializing [11333.920735] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11333.957790] Lustre: 436652:0:(lustre_peer.c:105:class_add_uuid()) 192.168.202.141@tcp: can't add NID 192.168.202.141@tcp130: rc = -75 [11333.961148] Lustre: 436652:0:(lustre_peer.c:105:class_add_uuid()) Skipped 17 previous similar messages [11333.983569] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11336.216469] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [11339.531661] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11342.397159] Lustre: server umount lustre-OST0000 complete [11343.855796] Lustre: 437677:0:(lustre_peer.c:105:class_add_uuid()) 192.168.202.141@tcp: can't add NID 192.168.202.141@tcp130: rc = -75 [11343.859335] Lustre: 437677:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [11343.873605] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11343.874472] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11343.877445] Lustre: Skipped 1 previous similar message [11348.960053] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11348.963052] Lustre: Skipped 2 previous similar messages [11353.056282] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11353.058788] Lustre: Skipped 1 previous similar message [11358.175082] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11358.175892] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11358.180440] Lustre: Skipped 1 previous similar message [11358.230742] Lustre: server umount lustre-MDT0000 complete [11359.545019] LustreError: 434894:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775227200 with bad export cookie 1643538000352083948 [11359.547980] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11359.549111] LustreError: 434894:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11359.669647] Lustre: server umount lustre-MDT0001 complete [11363.681830] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing unload_modules_local [11364.748973] Key type lgssc unregistered [11364.911711] LNet: 438421:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11364.915728] LNetError: 438421:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11364.918735] LNet: Removed LNI 192.168.202.141@tcp [11365.983172] LNet: Removed LNI 192.168.202.141@tcp100 [11365.984883] LNetError: 438421:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11365.987585] LNetError: 438421:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [11365.990625] LNet: Removed LNI 192.168.202.141@tcp101 [11367.007113] LNet: Removed LNI 192.168.202.141@tcp102 [11367.008793] LNetError: 438421:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11367.012657] LNetError: 438421:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [11367.015445] LNet: Removed LNI 192.168.202.141@tcp103 [11368.031116] LNet: Removed LNI 192.168.202.141@tcp104 [11369.055155] LNet: Removed LNI 192.168.202.141@tcp105 [11369.057226] LNetError: 438421:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11369.060977] LNetError: 438421:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [11369.064783] LNet: Removed LNI 192.168.202.141@tcp106 [11370.079176] LNet: Removed LNI 192.168.202.141@tcp107 [11371.103184] LNet: Removed LNI 192.168.202.141@tcp108 [11372.127148] LNet: Removed LNI 192.168.202.141@tcp109 [11373.151687] LNet: Removed LNI 192.168.202.141@tcp110 [11373.157120] LNetError: 438421:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11373.162030] LNetError: 438421:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [11373.168530] LNet: Removed LNI 192.168.202.141@tcp111 [11374.175133] LNet: Removed LNI 192.168.202.141@tcp112 [11375.199886] LNet: Removed LNI 192.168.202.141@tcp113 [11376.223138] LNet: Removed LNI 192.168.202.141@tcp114 [11377.247261] LNet: Removed LNI 192.168.202.141@tcp115 [11377.250217] LNet: Removed LNI 192.168.202.141@tcp116 [11377.252435] LNet: Removed LNI 192.168.202.141@tcp117 [11377.253881] LNet: Removed LNI 192.168.202.141@tcp118 [11377.255436] LNet: Removed LNI 192.168.202.141@tcp119 [11377.256716] LNet: Removed LNI 192.168.202.141@tcp120 [11377.258593] LNet: Removed LNI 192.168.202.141@tcp121 [11377.260225] LNet: Removed LNI 192.168.202.141@tcp122 [11377.261304] LNet: Removed LNI 192.168.202.141@tcp123 [11377.262898] LNet: Removed LNI 192.168.202.141@tcp124 [11377.264622] LNet: Removed LNI 192.168.202.141@tcp125 [11377.265741] LNet: Removed LNI 192.168.202.141@tcp126 [11377.266729] LNet: Removed LNI 192.168.202.141@tcp127 [11377.267771] LNet: Removed LNI 192.168.202.141@tcp128 [11377.268947] LNet: Removed LNI 192.168.202.141@tcp129 [11377.270650] LNet: Removed LNI 192.168.202.141@tcp130 [11377.272713] LNet: Removed LNI 192.168.202.141@tcp131 [11377.274048] LNet: Removed LNI 192.168.202.141@tcp132 [11377.275074] LNet: Removed LNI 192.168.202.141@tcp133 [11377.276238] LNet: Removed LNI 192.168.202.141@tcp134 [11377.284051] LNet: Removed LNI 192.168.202.141@tcp135 [11377.582795] Key type .llcrypt unregistered [11377.584605] Key type ._llcrypt unregistered [11387.024234] Key type ._llcrypt registered [11387.026157] Key type .llcrypt registered [11387.074571] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_hostid [11394.358253] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [11394.741301] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11394.780027] alg: No test for adler32 (adler32-zlib) [11395.671622] Lustre: Lustre: Build Version: 2.17.51_74_g8e7ba6b [11395.775946] LNet: Added LNI 192.168.202.141@tcp [8/256/0/180] [11397.375113] Key type lgssc registered [11397.858158] Lustre: Echo OBD driver; http://www.lustre.org/ [11410.835717] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 10:40:51 (1775227251) [11417.630547] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing unload_modules_local [11418.807508] Key type lgssc unregistered [11418.949469] LNet: 442427:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11418.953318] LNetError: 442427:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11418.964157] LNet: Removed LNI 192.168.202.141@tcp [11419.274751] Key type .llcrypt unregistered [11419.275851] Key type ._llcrypt unregistered [11426.793492] Key type ._llcrypt registered [11426.794877] Key type .llcrypt registered [11426.839677] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [11427.211940] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11427.277122] alg: No test for adler32 (adler32-zlib) [11428.147151] Lustre: Lustre: Build Version: 2.17.51_74_g8e7ba6b [11428.233610] LNet: Added LNI 192.168.202.141@tcp [8/256/0/180] [11429.815177] Key type lgssc registered [11430.280588] Lustre: Echo OBD driver; http://www.lustre.org/ [11434.184031] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11434.195915] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11435.298307] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11435.312561] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11435.354594] Lustre: lustre-MDT0000: new disk, initializing [11435.397523] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11435.406682] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11436.890614] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [11441.577510] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11441.628743] Lustre: 443599:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11441.669303] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11441.671389] Lustre: Skipped 1 previous similar message [11441.708260] Lustre: lustre-MDT0001: new disk, initializing [11441.748139] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11441.768278] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11441.772116] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11443.366686] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [11446.794590] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11448.266610] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11451.067437] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11451.208670] Lustre: lustre-OST0000: new disk, initializing [11451.211089] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11451.267674] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11453.540688] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [11457.020556] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11458.415044] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing lnet_if_list [11460.355312] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [11460.368723] LNet: Added LNI 192.168.202.141@tcp10 [8/256/0/180] [11461.121851] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11461.125602] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11461.167854] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11462.732699] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [11462.746368] LNet: Added LNI 192.168.202.141@tcp11 [8/256/0/180] [11464.399062] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [11464.415956] LNet: Added LNI 192.168.202.141@tcp12 [8/256/0/180] [11466.165229] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [11466.178752] LNet: Added LNI 192.168.202.141@tcp13 [8/256/0/180] [11467.870943] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [11467.884047] LNet: Added LNI 192.168.202.141@tcp14 [8/256/0/180] [11469.449814] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [11469.461698] LNet: Added LNI 192.168.202.141@tcp15 [8/256/0/180] [11471.197560] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [11471.210336] LNet: Added LNI 192.168.202.141@tcp16 [8/256/0/180] [11472.852517] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [11472.864481] LNet: Added LNI 192.168.202.141@tcp17 [8/256/0/180] [11474.512950] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [11474.525255] LNet: Added LNI 192.168.202.141@tcp18 [8/256/0/180] [11476.230203] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [11476.243174] LNet: Added LNI 192.168.202.141@tcp19 [8/256/0/180] [11478.014034] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [11478.027037] LNet: Added LNI 192.168.202.141@tcp20 [8/256/0/180] [11479.715283] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [11479.727402] LNet: Added LNI 192.168.202.141@tcp21 [8/256/0/180] [11481.489894] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [11481.503413] LNet: Added LNI 192.168.202.141@tcp22 [8/256/0/180] [11483.378213] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [11483.391922] LNet: Added LNI 192.168.202.141@tcp23 [8/256/0/180] [11485.288227] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [11485.301246] LNet: Added LNI 192.168.202.141@tcp24 [8/256/0/180] [11487.128800] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [11487.142328] LNet: Added LNI 192.168.202.141@tcp25 [8/256/0/180] [11489.114711] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [11489.129263] LNet: Added LNI 192.168.202.141@tcp26 [8/256/0/180] [11491.111498] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [11491.122549] LNet: Added LNI 192.168.202.141@tcp27 [8/256/0/180] [11493.008167] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [11493.020966] LNet: Added LNI 192.168.202.141@tcp28 [8/256/0/180] [11494.989293] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [11495.000602] LNet: Added LNI 192.168.202.141@tcp29 [8/256/0/180] [11496.919654] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [11496.931924] LNet: Added LNI 192.168.202.141@tcp30 [8/256/0/180] [11498.855992] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [11498.868136] LNet: Added LNI 192.168.202.141@tcp31 [8/256/0/180] [11500.883893] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [11500.896505] LNet: Added LNI 192.168.202.141@tcp32 [8/256/0/180] [11502.855599] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [11502.865833] LNet: Added LNI 192.168.202.141@tcp33 [8/256/0/180] [11504.858901] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [11504.870524] LNet: Added LNI 192.168.202.141@tcp34 [8/256/0/180] [11506.968906] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [11506.986332] LNet: Added LNI 192.168.202.141@tcp35 [8/256/0/180] [11509.444017] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [11509.456738] LNet: Added LNI 192.168.202.141@tcp36 [8/256/0/180] [11511.950240] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [11511.962137] LNet: Added LNI 192.168.202.141@tcp37 [8/256/0/180] [11514.325348] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [11514.337173] LNet: Added LNI 192.168.202.141@tcp38 [8/256/0/180] [11516.826226] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [11516.844260] LNet: Added LNI 192.168.202.141@tcp39 [8/256/0/180] [11519.724293] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [11519.740863] LNet: Added LNI 192.168.202.141@tcp40 [8/256/0/180] [11522.916858] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [11522.935727] LNet: Added LNI 192.168.202.141@tcp41 [8/256/0/180] [11525.751986] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [11525.765385] LNet: Added LNI 192.168.202.141@tcp42 [8/256/0/180] [11528.263869] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [11528.275883] LNet: Added LNI 192.168.202.141@tcp43 [8/256/0/180] [11530.905078] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [11530.917985] LNet: Added LNI 192.168.202.141@tcp44 [8/256/0/180] [11533.517213] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [11533.530066] LNet: Added LNI 192.168.202.141@tcp45 [8/256/0/180] [11536.621256] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [11536.636478] LNet: Added LNI 192.168.202.141@tcp46 [8/256/0/180] [11539.537121] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [11539.549535] LNet: Added LNI 192.168.202.141@tcp47 [8/256/0/180] [11542.365539] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [11542.378584] LNet: Added LNI 192.168.202.141@tcp48 [8/256/0/180] [11545.090234] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [11545.103911] LNet: Added LNI 192.168.202.141@tcp49 [8/256/0/180] [11548.297582] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [11548.311822] LNet: Added LNI 192.168.202.141@tcp50 [8/256/0/180] [11558.370513] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11558.372208] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11558.375217] Lustre: Skipped 2 previous similar messages [11558.377254] Lustre: Skipped 1 previous similar message [11562.646697] Lustre: server umount lustre-MDT0000 complete [11564.029038] LustreError: 444376:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775227405 with bad export cookie 3943030318942915818 [11564.032494] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11564.032838] LustreError: 444376:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11564.143170] Lustre: server umount lustre-MDT0001 complete [11571.760494] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [11575.188789] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11583.967148] Lustre: 442872:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775227409/real 1775227409] req@000000002a4ebd5c x1861460708903680/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1775227425 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11583.974654] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11585.759151] Lustre: 442873:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775227410/real 1775227410] req@00000000d38f9ec5 x1861460708903808/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1775227426 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11589.087113] Lustre: 442874:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775227414/real 1775227414] req@00000000b88a7fa4 x1861460708903936/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1775227430 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11589.087305] LustreError: 442871:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000f05b0739 x1861460708904704/t0(0) o250->MGC192.168.202.141@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11589.209474] LustreError: 445330:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11589.215940] LustreError: 445330:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [11589.227383] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11590.649777] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [11593.248261] LustreError: 445331:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11593.762882] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11593.895054] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11594.903085] Lustre: 442872:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775227420/real 1775227420] req@00000000fa65ab8c x1861460708904320/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1775227436 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11594.914428] Lustre: 442872:0:(client.c:2479:ptlrpc_expire_one_request()) Skipped 1 previous similar message [11594.917462] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [11595.365045] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [11597.289650] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11598.612321] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11609.568317] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11609.571755] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11609.577458] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11610.081692] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11612.128590] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11612.131613] Lustre: Skipped 2 previous similar messages [11612.911746] Lustre: server umount lustre-OST0000 complete [11614.687816] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11614.690134] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11614.695966] Lustre: Skipped 1 previous similar message [11614.698750] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11620.577926] Lustre: server umount lustre-MDT0000 complete [11621.927241] LustreError: 443591:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775227463 with bad export cookie 3943030318942917911 [11621.930449] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11621.931580] LustreError: 443591:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11622.053930] Lustre: server umount lustre-MDT0001 complete [11625.796228] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing unload_modules_local [11626.992059] Key type lgssc unregistered [11627.167231] LNet: 471268:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11627.170708] LNetError: 471268:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11627.173800] LNet: Removed LNI 192.168.202.141@tcp [11627.175834] LNet: Removed LNI 192.168.202.141@tcp10 [11627.177102] LNet: Removed LNI 192.168.202.141@tcp11 [11627.178161] LNet: Removed LNI 192.168.202.141@tcp12 [11627.179618] LNet: Removed LNI 192.168.202.141@tcp13 [11627.180686] LNet: Removed LNI 192.168.202.141@tcp14 [11627.181802] LNet: Removed LNI 192.168.202.141@tcp15 [11627.183225] LNet: Removed LNI 192.168.202.141@tcp16 [11627.184608] LNet: Removed LNI 192.168.202.141@tcp17 [11627.186319] LNet: Removed LNI 192.168.202.141@tcp18 [11627.187882] LNet: Removed LNI 192.168.202.141@tcp19 [11627.189379] LNet: Removed LNI 192.168.202.141@tcp20 [11627.190805] LNet: Removed LNI 192.168.202.141@tcp21 [11627.192243] LNet: Removed LNI 192.168.202.141@tcp22 [11627.193920] LNet: Removed LNI 192.168.202.141@tcp23 [11627.195447] LNet: Removed LNI 192.168.202.141@tcp24 [11627.196912] LNet: Removed LNI 192.168.202.141@tcp25 [11627.198466] LNet: Removed LNI 192.168.202.141@tcp26 [11627.199981] LNet: Removed LNI 192.168.202.141@tcp27 [11627.201626] LNet: Removed LNI 192.168.202.141@tcp28 [11627.203058] LNet: Removed LNI 192.168.202.141@tcp29 [11627.204460] LNet: Removed LNI 192.168.202.141@tcp30 [11627.206091] LNet: Removed LNI 192.168.202.141@tcp31 [11627.207419] LNet: Removed LNI 192.168.202.141@tcp32 [11627.208982] LNet: Removed LNI 192.168.202.141@tcp33 [11627.210344] LNet: Removed LNI 192.168.202.141@tcp34 [11627.211836] LNet: Removed LNI 192.168.202.141@tcp35 [11627.213352] LNet: Removed LNI 192.168.202.141@tcp36 [11627.214995] LNet: Removed LNI 192.168.202.141@tcp37 [11627.216803] LNet: Removed LNI 192.168.202.141@tcp38 [11627.218341] LNet: Removed LNI 192.168.202.141@tcp39 [11627.219924] LNet: Removed LNI 192.168.202.141@tcp40 [11627.221374] LNet: Removed LNI 192.168.202.141@tcp41 [11627.223117] LNet: Removed LNI 192.168.202.141@tcp42 [11627.224804] LNet: Removed LNI 192.168.202.141@tcp43 [11627.226428] LNet: Removed LNI 192.168.202.141@tcp44 [11627.228119] LNet: Removed LNI 192.168.202.141@tcp45 [11627.229717] LNet: Removed LNI 192.168.202.141@tcp46 [11627.231375] LNet: Removed LNI 192.168.202.141@tcp47 [11627.233049] LNet: Removed LNI 192.168.202.141@tcp48 [11627.234527] LNet: Removed LNI 192.168.202.141@tcp49 [11627.244365] LNet: Removed LNI 192.168.202.141@tcp50 [11627.535196] Key type .llcrypt unregistered [11627.536557] Key type ._llcrypt unregistered [11635.802586] Key type ._llcrypt registered [11635.803985] Key type .llcrypt registered [11635.852370] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [11636.304144] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11636.380482] alg: No test for adler32 (adler32-zlib) [11637.255420] Lustre: Lustre: Build Version: 2.17.51_74_g8e7ba6b [11637.362520] LNet: Added LNI 192.168.202.141@tcp [8/256/0/180] [11638.951174] Key type lgssc registered [11639.401555] Lustre: Echo OBD driver; http://www.lustre.org/ [11642.770312] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11642.779838] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11643.972521] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11645.501830] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [11649.034646] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11649.209646] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11650.774217] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [11652.856945] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11654.127253] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11656.531066] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11656.678037] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11658.733137] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [11660.763281] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11668.966457] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [11675.278762] loop0: detected capacity change from 0 to 204800000 [11675.375631] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11675.379610] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11675.384412] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11677.766348] loop0: detected capacity change from 0 to 204800000 [11677.884145] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11677.888247] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11677.896566] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11680.129885] loop0: detected capacity change from 0 to 204800000 [11682.433835] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11682.488327] Lustre: fs15246-OST0000: new disk, initializing [11682.491941] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [11682.524918] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11684.798390] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [11688.928201] loop1: detected capacity change from 0 to 204800000 [11690.962628] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11691.004199] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [11691.042281] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [11691.080572] Lustre: fs15246-MDT0000: new disk, initializing [11691.139761] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11691.149831] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11691.191465] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [11692.256063] Lustre: 476868:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@00000000b02367a6 x1861460928004224/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1775227549 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [11692.640134] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [11700.901346] LustreError: 476868:0:(fid_request.c:213:seq_client_alloc_seq()) cli-cli-fs15246-OST0000-osc-MDT0000: Cannot allocate new meta-sequence: rc = -5 [11700.905358] LustreError: 476868:0:(fid_request.c:255:seq_client_get_seq()) cli-cli-fs15246-OST0000-osc-MDT0000: Can't allocate new sequence: rc = -5 [11700.909209] LustreError: 476868:0:(osp_precreate.c:459:osp_precreate_rollover_new_seq()) fs15246-OST0000-osc-MDT0000: alloc fid error: rc = -5 [11701.222527] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [11706.339247] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [11707.028388] Lustre: server umount fs15246-MDT0000 complete [11710.314774] Lustre: server umount fs15246-OST0000 complete [11715.039671] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11715.041873] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11715.045518] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11716.064832] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11716.067499] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11721.695942] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11721.697997] Lustre: Skipped 3 previous similar messages [11726.816415] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11726.818812] Lustre: Skipped 1 previous similar message [11728.351064] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11728.409775] Lustre: server umount lustre-OST0000 complete [11731.423751] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11731.426643] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11737.056440] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11737.059587] Lustre: Skipped 3 previous similar messages [11744.223214] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11744.286324] Lustre: server umount lustre-MDT0000 complete [11745.596234] LustreError: 472471:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775227586 with bad export cookie 2394381945184207577 [11745.598298] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11745.599761] LustreError: 472471:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11745.726428] Lustre: server umount lustre-MDT0001 complete [11749.673123] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing unload_modules_local [11750.884841] Key type lgssc unregistered [11751.051429] LNet: 479932:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11751.054835] LNetError: 479932:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11751.069309] LNet: Removed LNI 192.168.202.141@tcp [11751.427381] Key type .llcrypt unregistered [11751.428791] Key type ._llcrypt unregistered [11760.125478] Key type ._llcrypt registered [11760.126827] Key type .llcrypt registered [11760.168390] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [11760.626150] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11760.637220] alg: No test for adler32 (adler32-zlib) [11761.513255] Lustre: Lustre: Build Version: 2.17.51_74_g8e7ba6b [11761.624445] LNet: Added LNI 192.168.202.141@tcp [8/256/0/180] [11763.215218] Key type lgssc registered [11763.650774] Lustre: Echo OBD driver; http://www.lustre.org/ [11781.181146] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_hostid [11785.771229] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [11800.236807] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11800.245933] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11801.360358] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11801.383119] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11801.423328] Lustre: lustre-MDT0000: new disk, initializing [11801.495528] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11801.510248] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11803.194343] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [11808.195679] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11808.243770] Lustre: 484317:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11808.274255] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11808.276573] Lustre: Skipped 1 previous similar message [11808.316935] Lustre: lustre-MDT0001: new disk, initializing [11808.354106] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11808.369115] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11808.372742] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11810.057193] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [11813.738351] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11815.273157] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11818.370278] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11818.487987] Lustre: lustre-OST0000: new disk, initializing [11818.490143] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11818.528364] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11820.786552] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [11824.189188] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11825.723726] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11826.177783] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11826.182782] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11826.230722] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11826.841399] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [11828.365051] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11828.448610] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11831.264529] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11831.265340] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11831.267399] LustreError: Skipped 1 previous similar message [11831.271943] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11831.274728] Lustre: Skipped 1 previous similar message [11835.237842] Lustre: server umount lustre-OST0000 complete [11838.944692] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11838.948612] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11838.953956] Lustre: Skipped 1 previous similar message [11838.956273] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11841.505054] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11841.505294] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11846.624306] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11846.626179] Lustre: Skipped 2 previous similar messages [11851.231101] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11851.296739] Lustre: server umount lustre-MDT0000 complete [11852.770180] LustreError: 484311:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775227693 with bad export cookie 15630469735690136176 [11852.772756] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11852.775716] LustreError: 484311:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11852.898941] Lustre: server umount lustre-MDT0001 complete [11862.327596] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_hostid [11866.133126] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [11878.059326] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 11552 sec ========= 10:48:38 (1775227718) [11878.706358] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 10:48:39 (1775227719) === [11879.967080] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 10:48:40 (1775227720) === [11895.230125] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing unload_modules_local [11896.403284] Key type lgssc unregistered [11896.540378] LNet: 492501:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11896.543237] LNetError: 492501:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11896.550347] LNet: Removed LNI 192.168.202.141@tcp [11896.874289] Key type .llcrypt unregistered [11896.875648] Key type ._llcrypt unregistered