[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 495443445 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2439 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22D5 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 002295 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2349 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23D9 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2411 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22d5-0xbffe2348] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22d4] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2349-0xbffe23d8] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23d9-0xbffe2410] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2411-0xbffe2438] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001013] APIC: Switch to symmetric I/O mode setup [ 0.003183] x2apic enabled [ 0.004008] Switched APIC routing to physical x2apic. [ 0.006009] kvm-guest: setup PV IPIs [ 0.008920] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.010018] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.011013] pid_max: default: 32768 minimum: 301 [ 0.012124] LSM: Security Framework initializing [ 0.013046] Yama: becoming mindful. [ 0.013817] SELinux: Initializing. [ 0.014057] *** VALIDATE selinux *** [ 0.021617] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026150] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027157] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028109] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029130] *** VALIDATE tmpfs *** [ 0.031330] *** VALIDATE proc *** [ 0.032202] *** VALIDATE cgroup *** [ 0.033010] *** VALIDATE cgroup2 *** [ 0.035254] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036158] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038033] Spectre V2 : User space: Vulnerable [ 0.039006] Speculative Store Bypass: Vulnerable [ 0.042373] debug: unmapping init [mem 0xffffffffb1c59000-0xffffffffb1c60fff] [ 0.044903] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045594] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046025] ... version: 2 [ 0.046769] ... bit width: 48 [ 0.047015] ... generic registers: 4 [ 0.047678] ... value mask: 0000ffffffffffff [ 0.048010] ... max period: 00007fffffffffff [ 0.049013] ... fixed-purpose events: 3 [ 0.049749] ... event mask: 000000070000000f [ 0.050286] rcu: Hierarchical SRCU implementation. [ 0.052369] smp: Bringing up secondary CPUs ... [ 0.053473] x86: Booting SMP configuration: [ 0.054013] .... node #0, CPUs: #1 #2 #3 [ 0.057073] smp: Brought up 1 node, 4 CPUs [ 0.058011] smpboot: Max logical packages: 1 [ 0.058749] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.138000] node 0 deferred pages initialised in 79ms [ 0.142160] devtmpfs: initialized [ 0.143198] x86/mm: Memory block size: 128MB [ 0.145770] gcov: version magic: 0x41383552 [ 0.146625] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.151150] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.154362] pinctrl core: initialized pinctrl subsystem [ 0.156179] [ 0.156826] ************************************************************* [ 0.160015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.163015] ** ** [ 0.165012] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.167011] ** ** [ 0.170015] ** This means that this kernel is built to expose internal ** [ 0.173016] ** IOMMU data structures, which may compromise security on ** [ 0.175011] ** your system. ** [ 0.177014] ** ** [ 0.180011] ** If you see this message and you are not debugging the ** [ 0.182011] ** kernel, report this immediately to your vendor! ** [ 0.185016] ** ** [ 0.187011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.189013] ************************************************************* [ 0.191730] NET: Registered protocol family 16 [ 0.194440] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.197063] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.199062] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.203008] cpuidle: using governor menu [ 0.204546] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.207832] PCI: Using configuration type 1 for base access [ 0.211269] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.224105] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.229033] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.234077] cryptd: max_cpu_qlen set to 1000 [ 0.236265] ACPI: Added _OSI(Module Device) [ 0.239019] ACPI: Added _OSI(Processor Device) [ 0.242017] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.245036] ACPI: Added _OSI(Processor Aggregator Device) [ 0.250021] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.255427] ACPI: Interpreter enabled [ 0.257063] ACPI: PM: (supports S0 S3 S4 S5) [ 0.259012] ACPI: Using IOAPIC for interrupt routing [ 0.260105] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.263397] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.273532] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.276049] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.279020] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.282080] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.287325] acpiphp: Slot [2] registered [ 0.289119] acpiphp: Slot [5] registered [ 0.291114] acpiphp: Slot [6] registered [ 0.292100] acpiphp: Slot [7] registered [ 0.293085] acpiphp: Slot [8] registered [ 0.294116] acpiphp: Slot [9] registered [ 0.296098] acpiphp: Slot [10] registered [ 0.298154] acpiphp: Slot [3] registered [ 0.299106] acpiphp: Slot [4] registered [ 0.301136] acpiphp: Slot [11] registered [ 0.303124] acpiphp: Slot [12] registered [ 0.304075] acpiphp: Slot [13] registered [ 0.305074] acpiphp: Slot [14] registered [ 0.306047] acpiphp: Slot [15] registered [ 0.307058] acpiphp: Slot [16] registered [ 0.308120] acpiphp: Slot [17] registered [ 0.311128] acpiphp: Slot [18] registered [ 0.313449] acpiphp: Slot [19] registered [ 0.316118] acpiphp: Slot [20] registered [ 0.318118] acpiphp: Slot [21] registered [ 0.321119] acpiphp: Slot [22] registered [ 0.323126] acpiphp: Slot [23] registered [ 0.325152] acpiphp: Slot [24] registered [ 0.328234] acpiphp: Slot [25] registered [ 0.330114] acpiphp: Slot [26] registered [ 0.333358] acpiphp: Slot [27] registered [ 0.335126] acpiphp: Slot [28] registered [ 0.338083] acpiphp: Slot [29] registered [ 0.339128] acpiphp: Slot [30] registered [ 0.341088] acpiphp: Slot [31] registered [ 0.342060] PCI host bridge to bus 0000:00 [ 0.344020] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.346021] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.348023] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.351027] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.354024] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.357024] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.359178] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.362134] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.365238] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.374018] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.379898] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.382017] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.385016] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.387016] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.389636] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.393978] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.397046] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.401993] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.408015] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.422016] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.426975] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.433487] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.443018] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.449014] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.463018] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.471000] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.478020] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.483015] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.503015] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.512383] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.519017] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.524017] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.535013] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.543168] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.551020] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.557015] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.577127] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.590997] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.601017] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.605016] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.622091] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.641864] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.650015] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.656016] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.669021] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.678914] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.681351] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.684472] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.686462] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.689185] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.693064] iommu: Default domain type: Passthrough [ 0.695408] SCSI subsystem initialized [ 0.696134] ACPI: bus type USB registered [ 0.697073] usbcore: registered new interface driver usbfs [ 0.699064] usbcore: registered new interface driver hub [ 0.700060] usbcore: registered new device driver usb [ 0.701157] pps_core: LinuxPPS API ver. 1 registered [ 0.703011] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.706063] PTP clock support registered [ 0.708203] EDAC MC: Ver: 3.0.0 [ 0.711109] PCI: Using ACPI for IRQ routing [ 0.713157] NetLabel: Initializing [ 0.715011] NetLabel: domain hash size = 128 [ 0.717011] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.719089] NetLabel: unlabeled traffic allowed by default [ 0.722046] vgaarb: loaded [ 0.723365] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.725011] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.740216] clocksource: Switched to clocksource kvm-clock [ 0.836072] VFS: Disk quotas dquot_6.6.0 [ 0.837723] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.840410] *** VALIDATE ramfs *** [ 0.841674] *** VALIDATE hugetlbfs *** [ 0.843322] pnp: PnP ACPI init [ 0.845572] pnp: PnP ACPI: found 6 devices [ 0.863136] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.866138] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.868437] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.870699] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.873058] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.875588] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.877815] NET: Registered protocol family 2 [ 0.880229] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.885328] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.889109] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.894458] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.897853] TCP: Hash tables configured (established 65536 bind 65536) [ 0.901525] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.905355] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.908708] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.912843] NET: Registered protocol family 1 [ 0.916237] RPC: Registered named UNIX socket transport module. [ 0.918241] RPC: Registered udp transport module. [ 0.919916] RPC: Registered tcp transport module. [ 0.921691] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.924469] NET: Registered protocol family 44 [ 0.926353] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.928821] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.931072] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.933783] PCI: CLS 0 bytes, default 64 [ 0.935616] Unpacking initramfs... [ 2.295145] debug: unmapping init [mem 0xffff971f7cc54000-0xffff971f7ffbffff] [ 2.299231] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.301627] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.304720] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.785468] Initialise system trusted keyrings [ 2.787191] Key type blacklist registered [ 2.790185] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.802990] zbud: loaded [ 2.806415] *** VALIDATE nfs *** [ 2.807830] *** VALIDATE nfs4 *** [ 2.810097] pstore: using deflate compression [ 2.813879] Platform Keyring initialized [ 2.906640] NET: Registered protocol family 38 [ 2.908534] Key type asymmetric registered [ 2.910123] Asymmetric key parser 'x509' registered [ 2.912349] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.915601] io scheduler mq-deadline registered [ 2.917585] io scheduler kyber registered [ 2.919283] io scheduler bfq registered [ 2.921125] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.924488] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.927561] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.930646] ACPI: Power Button [PWRF] [ 2.935817] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.944131] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.953825] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.959739] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.972461] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.999124] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.026790] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.032789] Non-volatile memory driver v1.3 [ 3.034354] Linux agpgart interface v0.103 [ 3.067688] virtio_blk virtio1: [vda] 134360 512-byte logical blocks (68.8 MB/65.6 MiB) [ 3.070627] vda: detected capacity change from 0 to 68792320 [ 3.090489] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.093123] vdb: detected capacity change from 0 to 1073741824 [ 3.111362] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.113142] vdc: detected capacity change from 0 to 2621440000 [ 3.125790] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.127626] vdd: detected capacity change from 0 to 2621440000 [ 3.138084] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.139893] vde: detected capacity change from 0 to 4294967296 [ 3.152257] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.154237] vdf: detected capacity change from 0 to 4294967296 [ 3.161140] libphy: Fixed MDIO Bus: probed [ 3.168787] usbcore: registered new interface driver usbserial_generic [ 3.171396] usbserial: USB Serial support registered for generic [ 3.173554] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.177420] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.178631] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.181407] mousedev: PS/2 mouse device common for all mice [ 3.183796] rtc_cmos 00:05: RTC can wake from S4 [ 3.186365] rtc_cmos 00:05: registered as rtc0 [ 3.186411] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.187517] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.194899] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.195464] intel_pstate: CPU model not supported [ 3.198486] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.202480] hid: raw HID events driver (C) Jiri Kosina [ 3.203752] usbcore: registered new interface driver usbhid [ 3.204913] usbhid: USB HID core driver [ 3.205938] drop_monitor: Initializing network drop monitor service [ 3.207335] Initializing XFRM netlink socket [ 3.208483] NET: Registered protocol family 10 [ 3.210506] Segment Routing with IPv6 [ 3.211419] NET: Registered protocol family 17 [ 3.212564] mpls_gso: MPLS GSO support [ 3.216546] RAS: Correctable Errors collector initialized. [ 3.217981] AVX version of gcm_enc/dec engaged. [ 3.219034] AES CTR mode by8 optimization enabled [ 3.277993] sched_clock: Marking stable (3277910158, 0)->(4122259610, -844349452) [ 3.281629] registered taskstats version 1 [ 3.283573] Loading compiled-in X.509 certificates [ 3.285639] zswap: loaded using pool lzo/zbud [ 3.308549] Key type big_key registered [ 3.319776] Key type encrypted registered [ 3.322231] ima: No TPM chip found, activating TPM-bypass! [ 3.324650] ima: Allocated hash algorithm: sha1 [ 3.326547] ima: No architecture policies found [ 3.328651] evm: Initialising EVM extended attributes: [ 3.330693] evm: security.selinux [ 3.331592] evm: security.ima [ 3.332361] evm: security.capability [ 3.333160] evm: HMAC attrs: 0x1 [ 3.335263] rtc_cmos 00:05: setting system clock to 2026-01-30 06:35:01 UTC (1769754901) [ 3.340423] debug: unmapping init [mem 0xffffffffb2c03000-0xffffffffb2dfffff] [ 3.343613] debug: unmapping init [mem 0xffffffffb1982000-0xffffffffb1c58fff] [ 3.354065] Write protecting the kernel read-only data: 28672k [ 3.356323] debug: unmapping init [mem 0xffffffffb0003000-0xffffffffb01fffff] [ 3.358027] debug: unmapping init [mem 0xffffffffb0914000-0xffffffffb09fffff] [ 3.382084] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.387561] systemd[1]: Detected virtualization kvm. [ 3.388681] systemd[1]: Detected architecture x86-64. [ 3.390027] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.413945] systemd[1]: No hostname configured. [ 3.415042] systemd[1]: Set hostname to . [ 3.416379] random: systemd: uninitialized urandom read (16 bytes read) [ 3.417835] systemd[1]: Initializing machine ID from random generator. [ 3.473608] random: ln: uninitialized urandom read (6 bytes read) [ 3.553292] random: systemd: uninitialized urandom read (16 bytes read) [ 3.555981] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.559922] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 3.564574] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Timers. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... Starting Create Volatile Files and Directories... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Initrd Root Device. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.177188] device-mapper: uevent: version 1.0.3 [ 4.178867] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.829461] random: fast init done [ 4.888427] virtio_net virtio0 ens2: renamed from eth0 [ 4.915431] scsi host0: ata_piix [ 4.926973] scsi host1: ata_piix [ 4.930027] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 4.934567] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.800547] dracut-initqueue[586]: RTNETLINK answers: File exists [ 9.839913] random: crng init done [ 9.841871] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.987870] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 13.840976] printk: systemd: 26 output lines suppressed due to ratelimiting [ 14.393113] SELinux: Disabled at runtime. [ 14.564455] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 14.580361] systemd[1]: Detected virtualization kvm. [ 14.586370] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 16.523996] systemd[1]: initrd-switch-root.service: Succeeded. [ 16.530992] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 16.551493] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 16.579610] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 16.587095] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 16.621289] systemd[1]: Starting Journal Service... Starting Journal Service... [ 16.651601] systemd[1]: Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice User and Session Slice. [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-sshd\x2dkeygen.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target Slices. Mounting POSIX Message Queue File System... [ OK ] Listening on udev Control Socket. [ OK ] Created slice system-getty.slice. Activating swap /dev/disk/by-label/SWAP... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ 17.061767] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting Huge Pages File System... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... Mounting Kernel Debug File System... Starting Remount Root and Kernel File Systems... [ OK ] Listening on Process Core Dump Socket. Starting Apply Kernel Variables... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... Starting Create Static Device Nodes in /dev... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 18.476558] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 19.770840] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 19.851654] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 20.703706] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 20.842672] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (8s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit) [*** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…only root support (10s / no limit) [ ***] A start job is running for Configur…only root support (10s / no limit)[ 27.056753] Key type dns_resolver registered [ **] A start job is running for Configur…only root support (11s / no limit) [ *] A start job is running for Configur…only root support (11s / no limit)[ 28.114439] NFS: Registering the id_resolver key type [ 28.119972] Key type id_resolver registered [ 28.125431] Key type id_legacy registered [ **] A start job is running for Configur…only root support (12s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started Login Service. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started OpenSSH server daemon. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg107-server login: [ 50.209331] hrtimer: interrupt took 2831808 ns [ 94.051588] libcfs: loading out-of-tree module taints kernel. [ 94.123326] Key type ._llcrypt registered [ 94.124946] Key type .llcrypt registered [ 94.264085] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_hostid [ 114.553428] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 117.004624] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 117.032292] alg: No test for adler32 (adler32-zlib) [ 118.841747] Lustre: Lustre: Build Version: 2.17.50_86_g2f7fa4d [ 120.260075] LNet: Added LNI 192.168.201.107@tcp [8/256/0/180] [ 122.113051] Key type lgssc registered [ 124.669442] Lustre: Echo OBD driver; http://www.lustre.org/ [ 145.534083] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 197.374591] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 212.427910] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 212.485851] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 213.701174] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 213.739990] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 213.855119] Lustre: lustre-MDT0000: new disk, initializing [ 213.989625] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 214.026871] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 218.485630] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 233.517535] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 233.658836] Lustre: 6516:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 233.703988] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 233.708412] Lustre: Skipped 1 previous similar message [ 233.786479] Lustre: lustre-MDT0001: new disk, initializing [ 233.899216] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 233.946719] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 233.977814] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 238.414541] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 243.442946] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 252.725425] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 253.107839] Lustre: lustre-OST0000: new disk, initializing [ 253.112226] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 253.202992] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 259.602619] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 259.614567] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 259.665412] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 260.068589] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 276.269144] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 276.420309] Lustre: lustre-OST0001: new disk, initializing [ 276.425338] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 276.523791] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 282.665046] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 282.681356] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 282.764965] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 283.867395] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 296.701876] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 307.646645] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 319.644349] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing check_logdir /tmp/testlogs/ [ 324.747168] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing yml_node [ 328.397949] Lustre: DEBUG MARKER: Client: 2.17.50.86 [ 330.949575] Lustre: DEBUG MARKER: MDS: 2.17.50.86 [ 333.482689] Lustre: DEBUG MARKER: OSS: 2.17.50.86 [ 335.191545] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Jan 30 01:40:32 EST 2026 [ 354.726201] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 356.418751] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 374.757811] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 374.758829] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 374.784330] Lustre: Skipped 1 previous similar message [ 379.183237] LustreError: 12215:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 379.295106] Lustre: server umount lustre-MDT0000 complete [ 383.260911] LustreError: 8426:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769755281 with bad export cookie 17512273887550818924 [ 383.262620] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 383.269202] LustreError: 8426:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 384.993922] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 384.995113] LustreError: 6523:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 385.012304] Lustre: Skipped 2 previous similar messages [ 385.023823] LustreError: 6523:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 385.042559] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 385.048919] Lustre: Skipped 4 previous similar messages [ 389.025874] LustreError: 6521:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 389.026251] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 389.063546] LustreError: 6521:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 389.078547] Lustre: Skipped 1 previous similar message [ 389.502410] LustreError: 12418:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 389.507509] LustreError: 12418:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 389.664155] Lustre: server umount lustre-MDT0001 complete [ 393.915468] LustreError: 12620:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 393.918290] LustreError: 12620:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 393.952867] Lustre: server umount lustre-OST0000 complete [ 398.155383] LustreError: 12820:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 398.161589] LustreError: 12820:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 398.272898] Lustre: server umount lustre-OST0001 complete [ 405.535671] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_hostid [ 417.580922] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 460.006131] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 460.300266] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 460.340819] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 460.447804] Lustre: lustre-MDT0000: new disk, initializing [ 460.557588] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 460.576652] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 467.851285] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 484.306714] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 484.490496] Lustre: 14993:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 484.632490] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 484.636853] Lustre: Skipped 1 previous similar message [ 484.813536] Lustre: lustre-MDT0001: new disk, initializing [ 484.986467] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 485.018550] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 485.028742] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 491.673274] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 501.075966] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 505.718793] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 514.735518] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 515.283816] Lustre: lustre-OST0000: new disk, initializing [ 515.289946] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 515.527591] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 517.116560] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 517.132955] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 517.430077] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 523.986800] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 532.483412] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 538.573507] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 538.839890] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 543.368462] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 543.580222] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 547.809666] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 547.827315] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 547.861406] Lustre: Skipped 1 previous similar message [ 547.876912] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 548.327646] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 553.453903] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 553.463173] Lustre: Skipped 3 previous similar messages [ 560.096140] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 560.165255] LustreError: 17928:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 560.175662] LustreError: 17928:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 560.353625] Lustre: server umount lustre-OST0000 complete [ 566.757159] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 566.774870] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 566.804263] Lustre: Skipped 1 previous similar message [ 566.811088] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 566.821076] Lustre: Skipped 2 previous similar messages [ 570.658375] LustreError: 18132:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 570.669125] LustreError: 18132:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 570.791386] Lustre: server umount lustre-MDT0000 complete [ 573.933210] LustreError: 15000:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 573.954122] LustreError: 15000:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 574.974819] LustreError: 15772:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769755473 with bad export cookie 17512273887550820457 [ 574.976937] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 574.983309] LustreError: 15772:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 575.557365] Lustre: server umount lustre-MDT0001 complete [ 581.003893] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 01:44:37 (1769755477) [ 584.560373] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 586.734609] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 01:44:43 (1769755483) [ 594.871754] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 595.418143] LustreError: 19005:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 595.522800] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 600.550253] LustreError: 19005:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 601.578295] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 610.790261] LustreError: 19004:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 610.832105] LustreError: 19004:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 613.753139] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 619.749551] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 626.130171] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 630.427361] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 640.278327] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 648.517861] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 654.595465] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 674.786097] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 674.791156] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 674.807543] Lustre: Skipped 1 previous similar message [ 674.817945] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 674.827972] Lustre: Skipped 2 previous similar messages [ 675.707050] LustreError: 21274:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 675.710802] LustreError: 21274:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 675.826942] Lustre: server umount lustre-OST0000 complete [ 680.933480] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 689.228621] Lustre: server umount lustre-MDT0000 complete [ 690.157837] LustreError: 19005:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 690.171313] LustreError: 19005:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 693.351374] LustreError: 20623:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769755591 with bad export cookie 17512273887550821332 [ 693.352504] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 693.360591] LustreError: 20623:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 693.844980] Lustre: server umount lustre-MDT0001 complete [ 704.108537] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing unload_modules_local [ 707.536969] Key type lgssc unregistered [ 707.869847] LNet: 22210:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 707.878591] LNetError: 22210:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 707.910124] LNet: Removed LNI 192.168.201.107@tcp [ 708.632185] Key type .llcrypt unregistered [ 708.633955] Key type ._llcrypt unregistered [ 732.959698] Key type ._llcrypt registered [ 732.962740] Key type .llcrypt registered [ 733.166377] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 734.827354] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 735.041362] alg: No test for adler32 (adler32-zlib) [ 736.282493] Lustre: Lustre: Build Version: 2.17.50_86_g2f7fa4d [ 736.537946] LNet: Added LNI 192.168.201.107@tcp [8/256/0/180] [ 738.392175] Key type lgssc registered [ 740.100919] Lustre: Echo OBD driver; http://www.lustre.org/ [ 749.374758] loop: module loaded [ 749.470543] loop0: detected capacity change from 0 to 204800000 [ 749.702122] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 749.719479] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 749.911652] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 756.328330] loop0: detected capacity change from 0 to 204800000 [ 756.614123] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 756.636527] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 756.668411] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 763.142646] loop0: detected capacity change from 0 to 204800000 [ 769.866843] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 769.918208] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 771.232937] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 771.307427] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 771.411469] Lustre: lustre-MDT0000: new disk, initializing [ 771.547701] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 771.567590] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 776.278700] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 786.217282] loop1: detected capacity change from 0 to 204800000 [ 792.979181] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 793.319666] Lustre: lustre-OST07c6: new disk, initializing [ 793.323101] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 793.334716] Lustre: Skipped 1 previous similar message [ 793.416522] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 799.194709] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 800.342876] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 800.353504] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 800.395852] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 803.580704] Lustre: 25869:0:(osd_internal.h:1455:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 803.589772] Lustre: 25869:0:(osd_handler.c:2093:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 803.595716] Lustre: 25869:0:(osd_handler.c:2100:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 803.600812] Lustre: 25869:0:(osd_handler.c:2110:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 803.606850] Lustre: 25869:0:(osd_handler.c:2117:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 803.615084] Lustre: 25869:0:(osd_handler.c:2124:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 819.699262] Lustre: 25869:0:(osd_internal.h:1455:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 819.703884] Lustre: 25869:0:(osd_handler.c:2093:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 819.710561] Lustre: 25869:0:(osd_handler.c:2100:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 819.714896] Lustre: 25869:0:(osd_handler.c:2110:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/4 [ 819.718616] Lustre: 25869:0:(osd_handler.c:2117:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 819.722355] Lustre: 25869:0:(osd_handler.c:2124:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 825.077941] Lustre: 24375:0:(osd_internal.h:1455:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 825.102223] Lustre: 24375:0:(osd_internal.h:1455:osd_trans_exec_op()) Skipped 1 previous similar message [ 825.117201] Lustre: 24375:0:(osd_handler.c:2093:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 825.131441] Lustre: 24375:0:(osd_handler.c:2093:osd_trans_dump_creds()) Skipped 1 previous similar message [ 825.137863] Lustre: 24375:0:(osd_handler.c:2100:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 825.155806] Lustre: 24375:0:(osd_handler.c:2100:osd_trans_dump_creds()) Skipped 1 previous similar message [ 825.173244] Lustre: 24375:0:(osd_handler.c:2110:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/0 [ 825.182862] Lustre: 24375:0:(osd_handler.c:2110:osd_trans_dump_creds()) Skipped 1 previous similar message [ 825.210226] Lustre: 24375:0:(osd_handler.c:2117:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 825.221325] Lustre: 24375:0:(osd_handler.c:2117:osd_trans_dump_creds()) Skipped 1 previous similar message [ 825.228018] Lustre: 24375:0:(osd_handler.c:2124:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 825.237477] Lustre: 24375:0:(osd_handler.c:2124:osd_trans_dump_creds()) Skipped 1 previous similar message [ 828.549663] Lustre: 25869:0:(osd_internal.h:1455:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 828.562753] Lustre: 25869:0:(osd_internal.h:1455:osd_trans_exec_op()) Skipped 1 previous similar message [ 828.569180] Lustre: 25869:0:(osd_handler.c:2093:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 828.581247] Lustre: 25869:0:(osd_handler.c:2093:osd_trans_dump_creds()) Skipped 1 previous similar message [ 828.599443] Lustre: 25869:0:(osd_handler.c:2100:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 828.608269] Lustre: 25869:0:(osd_handler.c:2100:osd_trans_dump_creds()) Skipped 1 previous similar message [ 828.616792] Lustre: 25869:0:(osd_handler.c:2110:osd_trans_dump_creds()) write: 2/23/0, punch: 0/0/0, quota 4/150/0 [ 828.627172] Lustre: 25869:0:(osd_handler.c:2110:osd_trans_dump_creds()) Skipped 1 previous similar message [ 828.639823] Lustre: 25869:0:(osd_handler.c:2117:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 828.645476] Lustre: 25869:0:(osd_handler.c:2117:osd_trans_dump_creds()) Skipped 1 previous similar message [ 828.653574] Lustre: 25869:0:(osd_handler.c:2124:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 828.662784] Lustre: 25869:0:(osd_handler.c:2124:osd_trans_dump_creds()) Skipped 1 previous similar message [ 837.065585] Lustre: 24375:0:(osd_internal.h:1455:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 837.077434] Lustre: 24375:0:(osd_internal.h:1455:osd_trans_exec_op()) Skipped 3 previous similar messages [ 837.085144] Lustre: 24375:0:(osd_handler.c:2093:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 837.091874] Lustre: 24375:0:(osd_handler.c:2093:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 837.100094] Lustre: 24375:0:(osd_handler.c:2100:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 837.107466] Lustre: 24375:0:(osd_handler.c:2100:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 837.120672] Lustre: 24375:0:(osd_handler.c:2110:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/0 [ 837.136904] Lustre: 24375:0:(osd_handler.c:2110:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 837.149120] Lustre: 24375:0:(osd_handler.c:2117:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 837.165893] Lustre: 24375:0:(osd_handler.c:2117:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 837.186149] Lustre: 24375:0:(osd_handler.c:2124:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 837.193940] Lustre: 24375:0:(osd_handler.c:2124:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 848.728447] Lustre: 24376:0:(osd_internal.h:1455:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 848.747565] Lustre: 24376:0:(osd_internal.h:1455:osd_trans_exec_op()) Skipped 5 previous similar messages [ 848.759267] Lustre: 24376:0:(osd_handler.c:2093:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 848.765600] Lustre: 24376:0:(osd_handler.c:2093:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 848.779139] Lustre: 24376:0:(osd_handler.c:2100:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 848.784797] Lustre: 24376:0:(osd_handler.c:2100:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 848.792215] Lustre: 24376:0:(osd_handler.c:2110:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/0 [ 848.798591] Lustre: 24376:0:(osd_handler.c:2110:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 848.804528] Lustre: 24376:0:(osd_handler.c:2117:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 848.810761] Lustre: 24376:0:(osd_handler.c:2117:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 848.816585] Lustre: 24376:0:(osd_handler.c:2124:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 848.822045] Lustre: 24376:0:(osd_handler.c:2124:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 897.517311] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 897.526873] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 897.547385] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 902.626051] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 907.232155] Lustre: lustre-OST07c6 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 907.273636] LustreError: 26560:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 907.625645] Lustre: server umount lustre-OST07c6 complete [ 917.885336] LustreError: 27065:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 917.898660] LustreError: 27065:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 918.064287] Lustre: server umount lustre-MDT0000 complete [ 934.034446] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing unload_modules_local [ 937.649719] Key type lgssc unregistered [ 938.033167] LNet: 27899:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 938.038787] LNetError: 27899:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 938.056853] LNet: Removed LNI 192.168.201.107@tcp [ 939.137971] Key type .llcrypt unregistered [ 939.144829] Key type ._llcrypt unregistered [ 958.825481] Key type ._llcrypt registered [ 958.827612] Key type .llcrypt registered [ 958.946869] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 960.340815] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 960.477891] alg: No test for adler32 (adler32-zlib) [ 961.849544] Lustre: Lustre: Build Version: 2.17.50_86_g2f7fa4d [ 962.136995] LNet: Added LNI 192.168.201.107@tcp [8/256/0/180] [ 963.953095] Key type lgssc registered [ 965.201372] Lustre: Echo OBD driver; http://www.lustre.org/ [ 972.935602] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 972.955973] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 974.588604] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 979.270988] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 984.045267] LustreError: 29148:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 984.086588] LustreError: 29148:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 988.259846] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 988.657562] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 993.119577] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 998.187435] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1002.022803] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1009.081056] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1009.439355] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1015.326665] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 1019.702281] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 1021.209553] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1039.337255] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1039.354468] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1039.366863] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1040.368153] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1040.369260] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1040.369311] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1042.761578] LustreError: 31592:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1043.242024] Lustre: server umount lustre-OST0000 complete [ 1050.082945] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1050.098373] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1050.114180] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1050.122014] Lustre: Skipped 1 previous similar message [ 1053.581804] LustreError: 31795:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1053.593597] LustreError: 31795:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1053.741573] Lustre: server umount lustre-MDT0000 complete [ 1055.714303] LustreError: 29148:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1055.735588] LustreError: 29148:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1057.331229] LustreError: 29129:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769755955 with bad export cookie 3260099468856772635 [ 1057.353073] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1057.463316] LustreError: 31997:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1057.468299] LustreError: 31997:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1057.612491] Lustre: server umount lustre-MDT0001 complete [ 1067.940284] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing unload_modules_local [ 1071.043831] Key type lgssc unregistered [ 1071.318768] LNet: 32529:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1071.338546] LNetError: 32529:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1071.365314] LNet: Removed LNI 192.168.201.107@tcp [ 1072.273808] Key type .llcrypt unregistered [ 1072.277035] Key type ._llcrypt unregistered [ 1091.863419] Key type ._llcrypt registered [ 1091.867415] Key type .llcrypt registered [ 1092.037904] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 1093.201022] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1093.257059] alg: No test for adler32 (adler32-zlib) [ 1094.447202] Lustre: Lustre: Build Version: 2.17.50_86_g2f7fa4d [ 1094.633073] LNet: Added LNI 192.168.201.107@tcp [8/256/0/180] [ 1096.232196] Key type lgssc registered [ 1097.764513] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1107.619205] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1107.647605] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1109.298635] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1115.062509] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 1119.726277] LustreError: 33777:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1119.742824] LustreError: 33777:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1124.839938] LustreError: 33777:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1125.503787] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1126.068982] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1131.319626] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 1138.156531] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1142.857896] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1151.357276] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1151.825381] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1154.860142] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1159.795198] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 1166.551405] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1183.909064] LustreError: 36042:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1184.183490] Lustre: server umount lustre-OST0000 complete [ 1191.907911] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1191.910458] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1191.955535] Lustre: Skipped 1 previous similar message [ 1191.982244] Lustre: Skipped 1 previous similar message [ 1197.026710] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1198.060948] LustreError: 36243:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1198.072631] LustreError: 36243:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1198.144939] Lustre: server umount lustre-MDT0000 complete [ 1202.145347] Lustre: 33088:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1769756084/real 1769756084] req@00000000c93cf6ae x1855723659099136/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1769756100 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1202.151282] LustreError: 34488:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1202.160728] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1202.185162] LustreError: 34488:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1202.274980] LustreError: 33759:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769756100 with bad export cookie 8517470391447085409 [ 1202.276073] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1202.279366] LustreError: 33759:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1202.415072] LustreError: 36445:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1202.428119] LustreError: 36445:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1202.669402] Lustre: server umount lustre-MDT0001 complete [ 1215.686866] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing unload_modules_local [ 1218.839397] Key type lgssc unregistered [ 1219.205362] LNet: 36978:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1219.222132] LNetError: 36978:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1219.244194] LNet: Removed LNI 192.168.201.107@tcp [ 1220.206739] Key type .llcrypt unregistered [ 1220.208830] Key type ._llcrypt unregistered [ 1247.517728] Key type ._llcrypt registered [ 1247.519628] Key type .llcrypt registered [ 1247.706815] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 1249.492123] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1249.716276] alg: No test for adler32 (adler32-zlib) [ 1251.034664] Lustre: Lustre: Build Version: 2.17.50_86_g2f7fa4d [ 1251.564892] LNet: Added LNI 192.168.201.107@tcp [8/256/0/180] [ 1253.376276] Key type lgssc registered [ 1254.861311] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1264.470996] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1264.493066] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1266.188423] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1271.069644] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 1276.385917] LustreError: 38375:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1276.398762] LustreError: 38375:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1280.972597] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1281.404834] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1286.487925] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 1293.086429] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1298.058326] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1305.335727] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1305.682420] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1311.869122] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1313.399070] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 1322.190772] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1330.547199] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.7@tcp (stopping) [ 1332.705776] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1332.711767] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1332.721332] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1332.747775] Lustre: Skipped 1 previous similar message [ 1335.609334] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.7@tcp (stopping) [ 1335.623314] Lustre: Skipped 2 previous similar messages [ 1336.452858] LustreError: 40620:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1336.727175] Lustre: server umount lustre-MDT0000 complete [ 1337.828380] LustreError: 38376:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1337.851585] LustreError: 38376:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1340.727276] LustreError: 38372:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1341.621354] LustreError: 38355:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769756239 with bad export cookie 18381254811044886024 [ 1341.635633] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1341.643881] LustreError: 38355:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1342.947369] LustreError: 39332:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1342.948954] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1343.012820] Lustre: Skipped 1 previous similar message [ 1343.021713] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1346.977549] LustreError: 39332:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1346.998907] LustreError: 39332:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1347.916203] LustreError: 40823:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1347.923181] LustreError: 40823:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1348.124850] Lustre: server umount lustre-MDT0001 complete [ 1369.546454] LustreError: 41048:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1369.552996] LustreError: 41048:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1369.741492] Lustre: server umount lustre-OST0000 complete [ 1394.297839] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing unload_modules_local [ 1397.730855] Key type lgssc unregistered [ 1398.158277] LNet: 41888:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1398.162431] LNetError: 41888:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1398.183863] LNet: Removed LNI 192.168.201.107@tcp [ 1399.177128] Key type .llcrypt unregistered [ 1399.180137] Key type ._llcrypt unregistered [ 1427.930784] Key type ._llcrypt registered [ 1427.932344] Key type .llcrypt registered [ 1428.029983] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 1428.953023] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1429.062336] alg: No test for adler32 (adler32-zlib) [ 1430.228487] Lustre: Lustre: Build Version: 2.17.50_86_g2f7fa4d [ 1430.466112] LNet: Added LNI 192.168.201.107@tcp [8/256/0/180] [ 1432.136189] Key type lgssc registered [ 1433.466388] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1443.134556] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1443.167480] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1444.757959] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1449.784639] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 1454.054867] LustreError: 43294:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1454.076634] LustreError: 43294:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1459.171604] LustreError: 43294:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1460.673651] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1461.208301] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1466.009938] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 1472.535337] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1477.805093] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1487.927016] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1488.328253] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1489.392923] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1494.570574] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 1501.699188] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1511.905400] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1511.918704] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1511.922403] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1511.958500] Lustre: Skipped 1 previous similar message [ 1513.220149] LustreError: 45539:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1513.515743] Lustre: server umount lustre-OST0000 complete [ 1532.388038] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1532.391173] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1532.398371] Lustre: Skipped 1 previous similar message [ 1532.408801] Lustre: Skipped 2 previous similar messages [ 1533.412748] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1533.417406] Lustre: Skipped 1 previous similar message [ 1538.523275] LustreError: 45916:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1538.530540] LustreError: 45916:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1538.531373] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1538.766426] Lustre: server umount lustre-MDT0000 complete [ 1543.608573] LustreError: 43276:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769756441 with bad export cookie 17227726723109197943 [ 1543.615863] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1543.629274] LustreError: 43276:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1549.905235] LustreError: 46118:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1549.934753] LustreError: 46118:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1550.123616] Lustre: server umount lustre-MDT0001 complete [ 1564.393483] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing unload_modules_local [ 1567.569706] Key type lgssc unregistered [ 1567.858826] LNet: 46665:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1567.865414] LNetError: 46665:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1568.941904] LNet: Removed LNI 192.168.201.107@tcp [ 1570.210588] Key type .llcrypt unregistered [ 1570.212782] Key type ._llcrypt unregistered [ 1597.417758] Key type ._llcrypt registered [ 1597.427933] Key type .llcrypt registered [ 1597.598889] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 1598.792530] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1598.911473] alg: No test for adler32 (adler32-zlib) [ 1600.058330] Lustre: Lustre: Build Version: 2.17.50_86_g2f7fa4d [ 1600.458697] LNet: Added LNI 192.168.201.107@tcp [8/256/0/180] [ 1602.216404] Key type lgssc registered [ 1603.952977] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1614.069174] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1614.102475] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1615.906789] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1621.673934] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 1625.086083] LustreError: 48067:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1625.096810] LustreError: 48067:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1630.179477] LustreError: 48067:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1631.884851] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1632.262617] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1637.443020] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 1644.983533] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1649.929250] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1658.484984] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1658.987054] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1660.035459] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1666.914511] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 1674.239207] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1684.128541] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1689.425322] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1702.395190] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-01-30 2h03m18s [ 1704.317904] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1707.869826] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.7@tcp (stopping) [ 1708.513655] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1708.531189] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1708.543314] Lustre: Skipped 1 previous similar message [ 1708.575429] Lustre: Skipped 2 previous similar messages [ 1709.604258] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1709.613124] Lustre: Skipped 3 previous similar messages [ 1715.017892] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.7@tcp (stopping) [ 1721.157551] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.7@tcp (stopping) [ 1721.164110] Lustre: Skipped 3 previous similar messages [ 1721.313237] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1721.465345] LustreError: 50632:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1721.830368] LustreError: 50223:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1721.861374] LustreError: 50223:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1721.878507] Lustre: server umount lustre-MDT0000 complete [ 1727.226628] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1727.296826] LustreError: 48063:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1733.474733] LustreError: 50223:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1733.501717] LustreError: 50223:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1735.864753] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1736.036270] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1736.345787] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1736.423221] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1737.416989] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1742.443502] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 1746.890777] Lustre: DEBUG MARKER: Wait for df (36152) ... [ 1748.914555] Lustre: DEBUG MARKER: done [ 1757.155929] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1757.166649] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1757.180466] Lustre: Skipped 1 previous similar message [ 1757.193788] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1758.185977] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1761.080286] LustreError: 51694:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1761.090618] LustreError: 51694:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1761.272112] Lustre: server umount lustre-OST0000 complete [ 1768.417657] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1768.445551] Lustre: Skipped 1 previous similar message [ 1775.586324] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1775.596503] Lustre: Skipped 7 previous similar messages [ 1780.192266] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1780.306402] LustreError: 51895:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1780.317505] LustreError: 51895:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1780.557739] Lustre: server umount lustre-MDT0000 complete [ 1784.739078] LustreError: 48062:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1784.767087] LustreError: 48062:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 1785.104509] LustreError: 48049:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769756683 with bad export cookie 7236255611505451779 [ 1785.109972] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1785.123869] LustreError: 48049:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1785.339890] LustreError: 52098:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1785.346289] LustreError: 52098:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1785.673478] Lustre: server umount lustre-MDT0001 complete [ 1799.308724] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing unload_modules_local [ 1802.772727] Key type lgssc unregistered [ 1803.152400] LNet: 52630:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1803.173094] LNetError: 52630:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1804.199642] LNet: Removed LNI 192.168.201.107@tcp [ 1805.287306] Key type .llcrypt unregistered [ 1805.289486] Key type ._llcrypt unregistered [ 1860.322672] Key type ._llcrypt registered [ 1860.327371] Key type .llcrypt registered [ 1860.494820] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 1862.409392] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1862.427118] alg: No test for adler32 (adler32-zlib) [ 1863.428281] Lustre: Lustre: Build Version: 2.17.50_86_g2f7fa4d [ 1863.725434] LNet: Added LNI 192.168.201.107@tcp [8/256/0/180] [ 1865.424520] Key type lgssc registered [ 1867.265286] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1879.343455] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1879.399705] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1880.810983] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1880.840817] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1881.051361] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1886.628107] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 1899.324611] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1899.525291] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1899.557197] Lustre: 54823:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1899.912035] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1906.085735] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 1912.256310] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1916.770362] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1924.813066] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1925.032319] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1925.214927] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1932.935955] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 1934.228887] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 1940.541852] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1948.996564] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-01-30 2h07m25s [ 1951.204986] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1960.239206] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1963.472456] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1963.755830] LustreError: 54831:0:(mdt_handler.c:2334:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1983.863359] Lustre: lustre-MDT0000: Client a12c6c45-56ce-4fc9-801f-47f66bb98560 (at 192.168.201.7@tcp) reconnecting [ 2003.768130] LustreError: 54831:0:(mdt_handler.c:2334:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 2006.979616] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 2010.082609] Lustre: DEBUG MARKER: done [ 2021.345509] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2021.352350] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2021.363295] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2021.373537] Lustre: Skipped 1 previous similar message [ 2023.485586] LustreError: 57862:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2023.703176] Lustre: server umount lustre-OST0000 complete [ 2031.588395] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2031.593597] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2031.605943] Lustre: Skipped 1 previous similar message [ 2031.630594] Lustre: Skipped 2 previous similar messages [ 2032.714453] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2032.725199] Lustre: Skipped 1 previous similar message [ 2034.623627] LustreError: 58065:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2034.634221] LustreError: 58065:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2034.828639] Lustre: server umount lustre-MDT0000 complete [ 2040.261978] LustreError: 56460:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769756938 with bad export cookie 7824375456708082863 [ 2040.267150] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2040.279069] LustreError: 56460:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2040.507464] LustreError: 58266:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2040.515522] LustreError: 58266:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2040.677369] Lustre: server umount lustre-MDT0001 complete [ 2054.736445] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing unload_modules_local [ 2058.515390] Key type lgssc unregistered [ 2058.927047] LNet: 58798:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2058.936405] LNetError: 58798:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2058.961969] LNet: Removed LNI 192.168.201.107@tcp [ 2060.271706] Key type .llcrypt unregistered [ 2060.273144] Key type ._llcrypt unregistered [ 2117.575959] Key type ._llcrypt registered [ 2117.578979] Key type .llcrypt registered [ 2117.753859] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 2119.611121] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2119.649937] alg: No test for adler32 (adler32-zlib) [ 2120.892951] Lustre: Lustre: Build Version: 2.17.50_86_g2f7fa4d [ 2121.369842] LNet: Added LNI 192.168.201.107@tcp [8/256/0/180] [ 2123.137675] Key type lgssc registered [ 2124.708286] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2128.609591] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2128.643558] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2130.100555] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2130.141718] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2130.382331] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2131.716084] Lustre: Failing over lustre-MDT0000 [ 2131.903548] LustreError: 61024:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2132.199699] Lustre: server umount lustre-MDT0000 complete [ 2140.308281] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 02:10:37 (1769757037) [ 2147.639958] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2148.321933] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2154.106208] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 2165.068571] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2165.234370] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2165.289137] Lustre: 61579:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2165.760717] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2171.442168] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 2178.400450] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2183.336664] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2191.820654] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2192.083759] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2192.332748] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2197.677690] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:291 to 0x280000401:321) [ 2199.679747] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 2206.328545] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2213.285197] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2222.049452] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2222.062058] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2222.078606] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2223.586439] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2223.589164] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2223.616512] Lustre: Skipped 2 previous similar messages [ 2228.714509] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2228.739955] Lustre: Skipped 1 previous similar message [ 2233.827672] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2233.850918] Lustre: Skipped 2 previous similar messages [ 2236.384203] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2236.515296] LustreError: 63952:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2236.525154] LustreError: 63952:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2236.660290] Lustre: server umount lustre-MDT0000 complete [ 2242.121253] LustreError: 61572:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769757140 with bad export cookie 7467135404194267733 [ 2242.125179] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2242.136316] LustreError: 61572:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2242.329337] LustreError: 64156:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2242.332694] LustreError: 64156:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2242.778643] Lustre: server umount lustre-MDT0001 complete [ 2247.726993] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2255.001647] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2260.258286] Lustre: 60504:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1769757142/real 1769757142] req@00000000ad3e98de x1855724735998592/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1769757158 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2260.306333] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2260.335125] Lustre: Skipped 1 previous similar message [ 2263.523519] Lustre: 60501:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1769757145/real 1769757145] req@00000000986bc732 x1855724735998720/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1769757161 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2265.568483] Lustre: 60502:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1769757147/real 1769757147] req@0000000070f6aac1 x1855724735998848/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1769757163 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2268.641900] Lustre: 60503:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1769757150/real 1769757150] req@0000000036e6738e x1855724735999104/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1769757166 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2268.944463] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2273.824173] Lustre: 60503:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1769757155/real 1769757155] req@00000000ada98ae6 x1855724735999232/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1769757171 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2287.024209] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x67a0981668be0b38 [ 2287.045720] Lustre: MGC192.168.201.107@tcp: Connection restored to 0@lo (at 0@lo) [ 2287.685748] LustreError: 63205:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2287.709064] LustreError: 63205:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2287.864857] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2288.060665] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2289.893628] LustreError: 63205:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2294.002782] LustreError: 63205:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2294.032221] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2294.159447] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 2299.364753] LustreError: 63205:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2304.520061] LustreError: 63205:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2305.954630] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2306.554673] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2311.660932] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2312.508507] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 2319.236861] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2323.962893] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2330.485731] LustreError: 66420:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2330.505709] LustreError: 66420:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2330.724708] Lustre: server umount lustre-MDT0000 complete [ 2332.130100] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2332.139472] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2332.151473] LustreError: 65018:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2332.151487] LustreError: 65018:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2332.207901] Lustre: Skipped 2 previous similar messages [ 2335.577718] LustreError: 61570:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769757233 with bad export cookie 7467135404194270008 [ 2335.600944] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2335.604774] LustreError: 61570:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2335.830397] LustreError: 66622:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2335.841465] LustreError: 66622:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2336.148638] Lustre: server umount lustre-MDT0001 complete [ 2343.241955] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2353.633865] Lustre: 60502:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1769757235/real 1769757235] req@000000000495e6be x1855724736025728/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1769757251 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2353.658194] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2354.839226] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2359.841185] LustreError: 60500:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000cfddead5 x1855724736026880/t0(0) o250->MGC192.168.201.107@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2360.376555] LustreError: 63204:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2360.388266] LustreError: 63204:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2360.502125] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2360.580605] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2365.574567] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 2372.081749] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2375.510787] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2381.305047] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2381.988743] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 2388.715678] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2393.991943] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2406.886108] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2406.890470] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2406.901783] Lustre: Skipped 2 previous similar messages [ 2406.902069] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2406.932623] Lustre: Skipped 2 previous similar messages [ 2408.940701] LustreError: 68879:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2408.946549] LustreError: 68879:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2409.201196] Lustre: server umount lustre-MDT0000 complete [ 2412.009320] LustreError: 67477:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2412.048996] LustreError: 67477:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 2414.444122] LustreError: 61572:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769757312 with bad export cookie 7467135404194271387 [ 2414.460645] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2415.196199] Lustre: server umount lustre-MDT0001 complete [ 2424.510478] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2429.079796] Lustre: DEBUG MARKER: files compared the same [ 2441.901836] LustreError: 69677:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2441.917695] LustreError: 69677:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 2442.225779] Lustre: server umount lustre-OST0000 complete [ 2461.195031] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing unload_modules_local [ 2464.357755] Key type lgssc unregistered [ 2464.705556] LNet: 70515:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2464.721161] LNetError: 70515:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2464.749532] LNet: Removed LNI 192.168.201.107@tcp [ 2465.713636] Key type .llcrypt unregistered [ 2465.718206] Key type ._llcrypt unregistered [ 2492.167544] Key type ._llcrypt registered [ 2492.169694] Key type .llcrypt registered [ 2492.341057] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 2493.758462] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2493.875855] alg: No test for adler32 (adler32-zlib) [ 2495.401906] Lustre: Lustre: Build Version: 2.17.50_86_g2f7fa4d [ 2495.700838] LNet: Added LNI 192.168.201.107@tcp [8/256/0/180] [ 2497.408414] Key type lgssc registered [ 2499.066527] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2508.687886] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2508.725839] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2510.411653] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2515.616653] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug +malloc all [ 2520.547316] LustreError: 71911:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2520.569220] LustreError: 71911:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2525.666181] LustreError: 71911:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2526.346750] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2526.920260] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2532.740476] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug +malloc all [ 2540.630378] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2544.940673] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2552.848996] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2553.554657] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2555.576699] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2561.027134] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug +malloc all [ 2567.748727] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2581.473387] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2581.491819] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2581.523585] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2582.504656] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2582.510433] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2582.525169] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2582.585142] Lustre: Skipped 1 previous similar message [ 2585.411340] LustreError: 74158:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2585.576551] Lustre: server umount lustre-OST0000 complete [ 2592.738505] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2592.740772] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2592.748301] Lustre: Skipped 1 previous similar message [ 2592.758741] Lustre: Skipped 1 previous similar message [ 2597.692133] LustreError: 74360:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2597.702794] LustreError: 74360:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2597.866086] LustreError: 74069:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2597.897483] LustreError: 74069:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2597.968162] Lustre: server umount lustre-MDT0000 complete [ 2601.964946] LustreError: 71911:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2601.991903] LustreError: 71911:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2602.721082] LustreError: 71891:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769757500 with bad export cookie 16122037301417011782 [ 2602.725182] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2602.735205] LustreError: 71891:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2602.900373] LustreError: 74562:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2602.906246] LustreError: 74562:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2603.379965] Lustre: server umount lustre-MDT0001 complete [ 2617.849270] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing unload_modules_local [ 2622.043419] Key type lgssc unregistered [ 2622.596730] LNet: 75096:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2622.604401] LNetError: 75096:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2622.620108] LNet: Removed LNI 192.168.201.107@tcp [ 2623.808451] Key type .llcrypt unregistered [ 2623.811885] Key type ._llcrypt unregistered [ 2649.767474] Key type ._llcrypt registered [ 2649.773652] Key type .llcrypt registered [ 2649.886407] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 2651.204928] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2651.275061] alg: No test for adler32 (adler32-zlib) [ 2652.394330] Lustre: Lustre: Build Version: 2.17.50_86_g2f7fa4d [ 2652.652883] LNet: Added LNI 192.168.201.107@tcp [8/256/0/180] [ 2654.312192] Key type lgssc registered [ 2656.326120] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2665.581852] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2665.615778] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2681.312479] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2686.433894] LustreError: 76468:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.107@tcp: failed processing log, type 4: rc = -110 [ 2713.122770] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2721.160980] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 2728.857451] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2738.355161] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2738.577161] Lustre: *** cfs_fail_loc=706, val=0*** [ 2739.155945] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2739.247333] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2744.298271] LustreError: 76494:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2744.328265] LustreError: 76494:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2744.593289] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 2748.400836] LustreError: 76494:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2748.429363] LustreError: 76494:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2753.508446] LustreError: 77441:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2753.538382] LustreError: 77441:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2754.721323] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2755.068610] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2760.187427] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 2767.125614] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2771.612065] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2775.013082] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2775.023450] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2775.044470] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2775.533274] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2775.536488] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2775.582559] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2780.645948] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2785.767577] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2785.776546] Lustre: Skipped 1 previous similar message [ 2788.320440] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2788.402812] LustreError: 78794:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 2788.493284] Lustre: server umount lustre-OST0000 complete [ 2796.005963] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2796.006876] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2796.028214] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2796.047551] Lustre: Skipped 1 previous similar message [ 2806.241625] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2806.248261] Lustre: Skipped 4 previous similar messages [ 2807.264794] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2807.413425] LustreError: 78997:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 2807.422422] LustreError: 78997:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2807.675418] Lustre: server umount lustre-MDT0000 complete [ 2811.362217] LustreError: 77445:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2811.377905] LustreError: 77445:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2812.035558] LustreError: 76475:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769757710 with bad export cookie 1798033761652153260 [ 2812.051208] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2812.053492] LustreError: 76475:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2812.215689] LustreError: 79200:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2812.238921] LustreError: 79200:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2812.495006] Lustre: server umount lustre-MDT0001 complete [ 2825.180756] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing unload_modules_local [ 2828.180287] Key type lgssc unregistered [ 2828.514123] LNet: 79731:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2828.521892] LNetError: 79731:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2829.547965] LNet: Removed LNI 192.168.201.107@tcp [ 2830.590689] Key type .llcrypt unregistered [ 2830.601037] Key type ._llcrypt unregistered [ 2856.537435] Key type ._llcrypt registered [ 2856.539533] Key type .llcrypt registered [ 2856.632827] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 2857.888674] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2858.056060] alg: No test for adler32 (adler32-zlib) [ 2859.196431] Lustre: Lustre: Build Version: 2.17.50_86_g2f7fa4d [ 2859.613524] LNet: Added LNI 192.168.201.107@tcp [8/256/0/180] [ 2861.376208] Key type lgssc registered [ 2863.051642] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2872.144083] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2872.179860] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2878.456676] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 2888.815566] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2889.338583] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2894.789846] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 2894.823415] LustreError: 81801:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2894.846254] LustreError: 81801:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2898.917668] LustreError: 81805:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2898.935113] LustreError: 81805:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2904.043398] LustreError: 81805:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2904.090736] LustreError: 81805:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2904.277523] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2904.715975] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2910.182757] LustreError: 81800:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2910.199329] LustreError: 81800:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2912.324496] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 2914.285751] LustreError: 81805:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2914.316648] LustreError: 81805:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2922.914101] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2922.941870] Lustre: MGS: Client 7565018c-10ac-4460-8e22-b741d1f02b8e (at 0@lo) reconnecting [ 2922.971221] LustreError: 81816:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2923.354758] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2923.464601] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2928.149605] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 2943.980260] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2943.986087] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2943.986692] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2944.002908] Lustre: Skipped 1 previous similar message [ 2948.701290] LustreError: 83783:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 2948.917918] Lustre: server umount lustre-OST0000 complete [ 2954.209454] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2954.213078] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2954.231356] Lustre: Skipped 1 previous similar message [ 2954.235549] Lustre: Skipped 1 previous similar message [ 2959.332560] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2959.336026] Lustre: Skipped 2 previous similar messages [ 2959.776041] LustreError: 83985:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2959.786590] LustreError: 83985:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2959.855814] Lustre: server umount lustre-MDT0000 complete [ 2964.112296] LustreError: 84186:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2964.118793] LustreError: 84186:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2964.445374] Lustre: server umount lustre-MDT0001 complete [ 2968.707925] LustreError: 84387:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2968.710585] LustreError: 84387:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2968.802866] Lustre: server umount MGS complete [ 2982.266391] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 02:24:39 (1769757879) [ 3014.330931] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_hostid [ 3024.563620] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 3064.915319] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3069.230966] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 3080.313543] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3080.622724] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 3080.711515] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3080.834863] Lustre: lustre-MDT0001: new disk, initializing [ 3080.935459] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3080.956551] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3080.964617] Lustre: 88895:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 3080.974557] LustreError: 88895:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 3085.752349] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 3097.899668] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3098.225217] Lustre: lustre-OST0000: new disk, initializing [ 3098.237856] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3098.350642] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3104.863793] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 3105.314620] Lustre: 89938:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@000000009eeb136f x1855725509845120/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1769758019 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 3113.561872] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3120.589155] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3120.607117] Lustre: MGS: Client effda1dd-f9c1-4120-9785-05bc518d7839 (at 0@lo) reconnecting [ 3120.684153] Lustre: 90576:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3120.698246] LustreError: 88881:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 3120.740581] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3120.850623] Lustre: lustre-MDT0000: new disk, initializing [ 3121.055445] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3121.077388] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3121.092940] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3122.146978] Lustre: 90603:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@00000000119ee425 x1855725509852544/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1769758036 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 3126.019204] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 3127.274729] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3131.247852] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3131.260529] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3131.345493] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 3156.962354] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3156.969252] LustreError: Skipped 1 previous similar message [ 3156.974482] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3156.992189] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3158.510946] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3163.618844] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3163.629377] Lustre: Skipped 3 previous similar messages [ 3171.296165] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3171.371673] LustreError: 91123:0:(obd_class.h:479:obd_check_dev()) Device 11 not setup [ 3171.377358] LustreError: 91123:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 3171.517291] Lustre: server umount lustre-OST0000 complete [ 3178.979113] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3178.980198] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3178.994848] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3178.994858] Lustre: Skipped 2 previous similar messages [ 3179.048934] Lustre: Skipped 2 previous similar messages [ 3182.256324] LustreError: 91328:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 3182.266308] LustreError: 91328:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3182.302470] Lustre: server umount lustre-MDT0000 complete [ 3184.112588] LustreError: 91012:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3184.145811] LustreError: 91012:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 3193.558746] Lustre: server umount lustre-MDT0001 complete [ 3198.633790] LustreError: 91733:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 3198.640871] LustreError: 91733:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 3198.690540] Lustre: server umount MGS complete [ 3211.192752] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 02:28:28 (1769758108) [ 3228.950972] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing unload_modules_local [ 3231.845994] Key type lgssc unregistered [ 3232.187665] LNet: 93071:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3232.193616] LNetError: 93071:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3232.207824] LNet: Removed LNI 192.168.201.107@tcp [ 3233.103415] Key type .llcrypt unregistered [ 3233.105943] Key type ._llcrypt unregistered [ 3250.345138] Key type ._llcrypt registered [ 3250.347092] Key type .llcrypt registered [ 3250.451897] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 3251.247662] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3251.499473] alg: No test for adler32 (adler32-zlib) [ 3252.568439] Lustre: Lustre: Build Version: 2.17.50_86_g2f7fa4d [ 3252.808069] LNet: Added LNI 192.168.201.107@tcp [8/256/0/180] [ 3254.608454] Key type lgssc registered [ 3255.970653] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3264.628960] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3264.668122] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3266.093948] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3270.278528] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 3276.259545] LustreError: 94281:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3276.297488] LustreError: 94281:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3278.964997] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3279.321370] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3283.214191] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 3288.983326] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3292.869277] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3298.852874] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3299.233185] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3304.782435] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 3309.701759] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3311.538917] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 3321.834272] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3321.834668] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3321.850123] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3321.867963] Lustre: Skipped 1 previous similar message [ 3321.898156] Lustre: Skipped 1 previous similar message [ 3326.948288] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3326.952151] Lustre: Skipped 1 previous similar message [ 3327.378960] LustreError: 96524:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3327.475357] Lustre: server umount lustre-OST0000 complete [ 3332.068304] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3332.085589] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3332.090719] Lustre: Skipped 1 previous similar message [ 3337.188120] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3337.206139] Lustre: Skipped 1 previous similar message [ 3342.305656] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3342.308206] Lustre: Skipped 1 previous similar message [ 3345.376199] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3345.436067] LustreError: 96727:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3345.449461] LustreError: 96727:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3345.554357] Lustre: server umount lustre-MDT0000 complete [ 3347.431484] LustreError: 95003:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3347.450045] LustreError: 95003:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3349.498963] LustreError: 94261:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769758247 with bad export cookie 6567341432052774505 [ 3349.503354] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3349.508692] LustreError: 94261:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3349.660718] LustreError: 96929:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 3349.668805] LustreError: 96929:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3349.933904] Lustre: server umount lustre-MDT0001 complete [ 3360.518386] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing unload_modules_local [ 3363.486731] Key type lgssc unregistered [ 3363.764291] LNet: 97462:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3363.773780] LNetError: 97462:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3363.790365] LNet: Removed LNI 192.168.201.107@tcp [ 3364.599804] Key type .llcrypt unregistered [ 3364.603136] Key type ._llcrypt unregistered [ 3378.590760] Key type ._llcrypt registered [ 3378.593345] Key type .llcrypt registered [ 3378.692245] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 3379.839715] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3379.855479] alg: No test for adler32 (adler32-zlib) [ 3380.914654] Lustre: Lustre: Build Version: 2.17.50_86_g2f7fa4d [ 3381.103579] LNet: Added LNI 192.168.201.107@tcp [8/256/0/180] [ 3382.760200] Key type lgssc registered [ 3384.084431] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3389.947985] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3389.948167] LustreError: 98506:0:(tgt_mount.c:2535:server_fill_super()) cfs_race id 716 sleeping [ 3389.951796] Lustre: Skipped 1 previous similar message [ 3389.951814] LustreError: 98505:0:(tgt_mount.c:2535:server_fill_super()) cfs_fail_race id 716 waking [ 3389.963778] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3389.964614] LustreError: 98506:0:(tgt_mount.c:2535:server_fill_super()) cfs_fail_race id 716 awake: rc=5000 [ 3389.975874] LustreError: 98506:0:(tgt_mount.c:2547:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3389.986123] LustreError: 98506:0:(super25.c:190:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3391.385298] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3397.313375] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3397.642323] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3401.267109] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 3405.801059] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing lsmod [ 3408.201682] LustreError: 99676:0:(tgt_mount.c:2535:server_fill_super()) cfs_race id 716 sleeping [ 3408.234344] LustreError: 99679:0:(tgt_mount.c:2535:server_fill_super()) cfs_fail_race id 716 waking [ 3408.239195] LustreError: 99676:0:(tgt_mount.c:2535:server_fill_super()) cfs_fail_race id 716 awake: rc=4966 [ 3408.245448] LustreError: 99676:0:(tgt_mount.c:2547:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3408.256714] LustreError: 99676:0:(super25.c:190:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3408.271202] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3408.619817] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3412.965881] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3412.970763] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3412.982799] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3413.989375] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3413.994771] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3414.005729] Lustre: Skipped 1 previous similar message [ 3414.019206] Lustre: Skipped 2 previous similar messages [ 3416.869267] LustreError: 99866:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3416.936670] Lustre: server umount lustre-MDT0000 complete [ 3419.107470] LustreError: 98544:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3419.116946] LustreError: 98544:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 3420.367791] LustreError: 98517:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769758318 with bad export cookie 12546929741787626670 [ 3420.372109] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3420.380726] LustreError: 98517:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3420.407191] LustreError: 98540:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3420.416408] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3420.438352] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3420.568922] LustreError: 100070:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 3420.578254] LustreError: 100070:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3420.812912] Lustre: server umount lustre-MDT0001 complete [ 3424.502604] LustreError: 100272:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 3424.510293] LustreError: 100272:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3424.608315] Lustre: server umount lustre-OST0000 complete [ 3431.745533] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3432.041437] LustreError: 100747:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3432.098496] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3435.923707] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 3437.542581] LustreError: 100747:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3442.661740] LustreError: 100747:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3443.777795] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3444.148427] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3448.628907] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 3454.530430] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3458.127585] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3464.778102] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3465.120713] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3471.603587] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 3477.430068] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3478.002953] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3484.649212] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3484.652981] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3484.676525] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3487.730575] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3489.603893] LustreError: 102988:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3489.607617] LustreError: 102988:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3489.803746] Lustre: server umount lustre-OST0000 complete [ 3495.403804] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3495.415171] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3495.420811] Lustre: Skipped 1 previous similar message [ 3495.426845] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3495.430629] Lustre: Skipped 2 previous similar messages [ 3499.595877] LustreError: 103190:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3499.604589] LustreError: 103190:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3499.674260] Lustre: server umount lustre-MDT0000 complete [ 3503.073280] LustreError: 100747:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3503.091415] LustreError: 100747:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3503.255325] LustreError: 100728:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769758401 with bad export cookie 12546929741787627363 [ 3503.263534] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3503.267320] LustreError: 100728:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3503.602941] Lustre: server umount lustre-MDT0001 complete [ 3514.541378] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing unload_modules_local [ 3517.134703] Key type lgssc unregistered [ 3517.428704] LNet: 103926:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3517.439842] LNetError: 103926:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3517.461093] LNet: Removed LNI 192.168.201.107@tcp [ 3518.438843] Key type .llcrypt unregistered [ 3518.440586] Key type ._llcrypt unregistered [ 3540.198976] Key type ._llcrypt registered [ 3540.201284] Key type .llcrypt registered [ 3540.299865] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 3541.526119] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3541.550249] alg: No test for adler32 (adler32-zlib) [ 3542.497278] Lustre: Lustre: Build Version: 2.17.50_86_g2f7fa4d [ 3542.690117] LNet: Added LNI 192.168.201.107@tcp [8/256/0/180] [ 3544.337418] Key type lgssc registered [ 3545.371582] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3551.789351] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3551.815030] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3553.207146] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3557.123941] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 3563.495490] LustreError: 105324:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3563.513137] LustreError: 105324:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3564.779411] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3565.289628] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3569.350996] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 3575.345410] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3579.060792] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3586.002848] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3586.466118] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3592.519249] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 3598.370771] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3599.811188] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 3607.166524] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3610.380938] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3615.204050] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3615.204508] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3615.219871] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3615.234475] Lustre: Skipped 1 previous similar message [ 3620.323481] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3620.334407] Lustre: Skipped 2 previous similar messages [ 3620.472135] LustreError: 107666:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3620.566291] Lustre: server umount lustre-OST0000 complete [ 3625.444092] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3625.452434] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3625.455355] Lustre: Skipped 1 previous similar message [ 3630.561725] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3630.570564] Lustre: Skipped 1 previous similar message [ 3635.681382] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3638.240164] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3638.334742] LustreError: 107871:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3638.339413] LustreError: 107871:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3638.492941] Lustre: server umount lustre-MDT0000 complete [ 3640.802104] LustreError: 105320:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3640.819239] LustreError: 105320:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3641.569616] LustreError: 106039:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769758539 with bad export cookie 12085554644802593769 [ 3641.579493] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3641.582041] LustreError: 106039:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3641.684929] LustreError: 108074:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 3641.689210] LustreError: 108074:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3641.835301] Lustre: server umount lustre-MDT0001 complete [ 3650.379515] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing unload_modules_local [ 3652.696254] Key type lgssc unregistered [ 3652.927880] LNet: 108605:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3652.933893] LNetError: 108605:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3653.990768] LNet: Removed LNI 192.168.201.107@tcp [ 3654.743391] Key type .llcrypt unregistered [ 3654.746101] Key type ._llcrypt unregistered [ 3668.099943] Key type ._llcrypt registered [ 3668.101560] Key type .llcrypt registered [ 3668.195709] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 3669.016635] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3669.158859] alg: No test for adler32 (adler32-zlib) [ 3670.184459] Lustre: Lustre: Build Version: 2.17.50_86_g2f7fa4d [ 3670.412613] LNet: Added LNI 192.168.201.107@tcp [8/256/0/180] [ 3672.080555] Key type lgssc registered [ 3673.147910] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3679.887346] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3679.904282] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3681.295068] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3684.679713] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 3691.491678] LustreError: 109787:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3691.526096] LustreError: 109787:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3692.643941] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3692.971220] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3696.223938] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 3700.474278] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3703.356376] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3708.790194] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3709.077740] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3713.571856] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 3718.019904] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3726.546612] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3737.058402] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3737.063555] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3737.065756] Lustre: Skipped 1 previous similar message [ 3737.080950] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3742.182566] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3742.185420] Lustre: Skipped 1 previous similar message [ 3742.787152] LustreError: 112050:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3743.034339] Lustre: server umount lustre-OST0000 complete [ 3747.298607] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3747.309986] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3747.315727] Lustre: Skipped 1 previous similar message [ 3752.419274] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3752.431931] Lustre: Skipped 1 previous similar message [ 3752.833791] LustreError: 112252:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3752.841573] LustreError: 112252:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3752.937158] Lustre: server umount lustre-MDT0000 complete [ 3756.319589] LustreError: 109767:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769758654 with bad export cookie 3487951198056001954 [ 3756.321760] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3756.341469] LustreError: 109767:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3756.483192] LustreError: 112453:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 3756.490470] LustreError: 112453:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3756.719818] Lustre: server umount lustre-MDT0001 complete [ 3765.651461] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing unload_modules_local [ 3767.872534] Key type lgssc unregistered [ 3768.143801] LNet: 112985:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3768.152431] LNetError: 112985:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3769.196249] LNet: Removed LNI 192.168.201.107@tcp [ 3769.856681] Key type .llcrypt unregistered [ 3769.860221] Key type ._llcrypt unregistered [ 3788.477371] Key type ._llcrypt registered [ 3788.481310] Key type .llcrypt registered [ 3788.545617] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 3789.301612] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3789.491375] alg: No test for adler32 (adler32-zlib) [ 3790.516831] Lustre: Lustre: Build Version: 2.17.50_86_g2f7fa4d [ 3790.684876] LNet: Added LNI 192.168.201.107@tcp [8/256/0/180] [ 3792.368192] Key type lgssc registered [ 3793.450549] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3800.130818] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3800.147477] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3801.538228] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3806.067288] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 3811.812518] LustreError: 114360:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3811.840706] LustreError: 114360:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3814.756877] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3815.050546] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3819.003063] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 3823.662830] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3826.561508] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3831.702204] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3832.076603] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3837.340783] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 3842.159099] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3846.484230] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3852.828970] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3855.953942] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3855.958017] Lustre: Skipped 2 previous similar messages [ 3859.384309] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3859.390994] Lustre: Skipped 2 previous similar messages [ 3861.068287] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3861.090904] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3868.251791] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.201.7@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3868.260700] Lustre: Skipped 2 previous similar messages [ 3868.596176] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.201.7@tcp 192.168.0.[2,10]@tcp [ 3875.768336] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.201.7/24@tcp in log lustre-MDT0000 [ 3875.776461] Lustre: Skipped 2 previous similar messages [ 3884.076941] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.7/24@tcp [ 3884.083300] Lustre: Skipped 1 previous similar message [ 3891.184964] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3891.188659] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3891.202126] LustreError: Skipped 1 previous similar message [ 3891.257595] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3891.525273] LustreError: 117907:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3891.684976] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3891.690391] LustreError: Skipped 2 previous similar messages [ 3891.696322] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3891.708690] LustreError: 115953:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3891.862730] Lustre: server umount lustre-OST0000 complete [ 3896.801879] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3896.811484] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3896.823978] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3897.827149] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3897.842704] Lustre: Skipped 1 previous similar message [ 3901.637660] LustreError: 118108:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3901.646706] LustreError: 118108:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3901.821072] Lustre: server umount lustre-MDT0000 complete [ 3902.950543] LustreError: 114361:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3902.978596] LustreError: 114361:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3906.013314] LustreError: 114341:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769758804 with bad export cookie 6090152226318843103 [ 3906.019455] LustreError: 114341:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3906.021811] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3906.129380] LustreError: 118310:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 3906.137332] LustreError: 118310:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3906.351857] Lustre: server umount lustre-MDT0001 complete [ 3918.102349] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing unload_modules_local [ 3920.726937] Key type lgssc unregistered [ 3921.081273] LNet: 118841:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3921.089613] LNetError: 118841:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3921.099697] LNet: Removed LNI 192.168.201.107@tcp [ 3921.844111] Key type .llcrypt unregistered [ 3921.846625] Key type ._llcrypt unregistered [ 3946.296183] Key type ._llcrypt registered [ 3946.300167] Key type .llcrypt registered [ 3946.428689] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 3947.496211] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3947.516783] alg: No test for adler32 (adler32-zlib) [ 3948.463169] Lustre: Lustre: Build Version: 2.17.50_86_g2f7fa4d [ 3948.647289] LNet: Added LNI 192.168.201.107@tcp [8/256/0/180] [ 3950.272377] Key type lgssc registered [ 3951.212833] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3957.329783] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3957.345897] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3958.859263] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3958.888541] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.201.7/24@tcp [ 3958.901276] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3962.602368] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 3969.414929] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3969.507547] LustreError: 120362:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3969.531609] LustreError: 120362:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3969.719809] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3969.741268] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.7/24@tcp [ 3969.796928] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3973.805222] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 3977.882216] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3980.619775] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3985.060088] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3985.301759] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3989.500317] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 3993.173373] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4004.789618] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 4012.990102] LustreError: 122680:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4013.114627] Lustre: server umount lustre-OST0000 complete [ 4016.102320] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4016.111906] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4016.122340] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4020.194852] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4020.195840] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4025.313304] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4025.314898] Lustre: Skipped 2 previous similar messages [ 4026.353686] LustreError: 122881:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4026.363476] LustreError: 122881:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4026.425800] Lustre: server umount lustre-MDT0000 complete [ 4029.457714] LustreError: 120344:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769758927 with bad export cookie 9710589739742087071 [ 4029.462901] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4029.466244] LustreError: 120344:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4039.657674] LustreError: 123083:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 4039.666130] LustreError: 123083:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4039.799326] Lustre: server umount lustre-MDT0001 complete [ 4047.054281] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing unload_modules_local [ 4048.796756] Key type lgssc unregistered [ 4048.995765] LNet: 123616:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4048.999062] LNetError: 123616:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4049.010360] LNet: Removed LNI 192.168.201.107@tcp [ 4049.522418] Key type .llcrypt unregistered [ 4049.525397] Key type ._llcrypt unregistered [ 4069.894325] Key type ._llcrypt registered [ 4069.896195] Key type .llcrypt registered [ 4069.968915] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_hostid [ 4082.287507] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 4083.343415] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4083.370380] alg: No test for adler32 (adler32-zlib) [ 4084.356669] Lustre: Lustre: Build Version: 2.17.50_86_g2f7fa4d [ 4084.593535] LNet: Added LNI 192.168.201.107@tcp [8/256/0/180] [ 4086.232373] Key type lgssc registered [ 4087.036779] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4114.704215] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4114.724475] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4115.904698] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4115.951407] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4115.999873] Lustre: lustre-MDT0000: new disk, initializing [ 4116.065222] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4116.079184] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4118.900863] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 4126.828264] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4126.917487] Lustre: 126958:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4126.964551] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4126.967629] Lustre: Skipped 1 previous similar message [ 4127.027440] Lustre: lustre-MDT0001: new disk, initializing [ 4127.124940] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4127.162259] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4127.174863] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4130.459163] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 4137.672199] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4140.643813] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4146.337986] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4146.613597] Lustre: lustre-OST0000: new disk, initializing [ 4146.616712] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4146.711115] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4152.264485] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 4153.406827] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4153.412702] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4153.526531] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4158.352311] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4162.320268] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4162.535904] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4165.715914] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4165.950745] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4169.188324] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4169.206477] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4169.249576] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4169.256648] Lustre: Skipped 1 previous similar message [ 4173.632238] LustreError: 129888:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4173.760303] Lustre: server umount lustre-OST0000 complete [ 4178.403877] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4178.408170] LustreError: Skipped 1 previous similar message [ 4178.411298] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4178.420027] Lustre: Skipped 1 previous similar message [ 4178.424702] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4179.426904] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4179.429444] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4179.445062] Lustre: Skipped 1 previous similar message [ 4182.375768] LustreError: 130089:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4182.380462] LustreError: 130089:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4182.481668] Lustre: server umount lustre-MDT0000 complete [ 4184.998936] LustreError: 126951:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769759083 with bad export cookie 17697837555507763553 [ 4185.008869] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4185.011485] LustreError: 126951:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4185.109477] LustreError: 130291:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4185.113787] LustreError: 130291:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4185.249916] Lustre: server umount lustre-MDT0001 complete [ 4191.817565] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4192.231164] LustreError: 130767:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4192.242241] LustreError: 130767:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4192.279248] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4195.367557] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 4197.350936] LustreError: 130767:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4202.068196] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4202.329347] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4205.796248] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 4209.763682] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4212.507936] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4217.665653] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4217.997281] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4222.063504] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 4225.952669] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4228.736915] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4228.863114] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4232.030310] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4232.185561] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4239.848769] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4240.019371] Lustre: lustre-OST0001: new disk, initializing [ 4240.025708] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4240.070275] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4244.955533] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 4246.114263] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4246.123404] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4246.229067] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4251.838456] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4252.007315] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4255.562543] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4255.698819] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4258.788300] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9a24d32da000.ost_server_uuid 50 [ 4260.033379] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9a24d32da000.ost_server_uuid in FULL state after 0 sec [ 4266.976922] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4266.979046] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4266.992800] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4266.995772] Lustre: Skipped 1 previous similar message [ 4267.004429] Lustre: Skipped 1 previous similar message [ 4271.923379] LustreError: 135004:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 4271.928193] LustreError: 135004:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4271.993878] Lustre: server umount lustre-OST0001 complete [ 4272.116170] LustreError: 132358:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4272.126169] LustreError: 132358:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4275.682250] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4275.686503] LustreError: Skipped 1 previous similar message [ 4275.691070] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4275.702887] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4277.229684] LustreError: 134173:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4277.249015] LustreError: 134173:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4280.631996] LustreError: 135205:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4280.636988] LustreError: 135205:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4280.719451] Lustre: server umount lustre-OST0000 complete [ 4284.390580] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4284.398060] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4284.409317] Lustre: Skipped 1 previous similar message [ 4284.415254] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4284.419316] Lustre: Skipped 2 previous similar messages [ 4289.578734] LustreError: 135406:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 4289.587056] LustreError: 135406:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4291.649765] Lustre: server umount lustre-MDT0000 complete [ 4292.579419] LustreError: 131548:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4292.593303] LustreError: 131548:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4294.022716] LustreError: 130791:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769759192 with bad export cookie 17697837555507764435 [ 4294.025360] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4294.028378] LustreError: 130791:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4294.210381] Lustre: server umount lustre-MDT0001 complete [ 4306.619753] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing unload_modules_local [ 4308.512354] Key type lgssc unregistered [ 4308.691620] LNet: 136597:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4308.698439] LNetError: 136597:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4308.710852] LNet: Removed LNI 192.168.201.107@tcp [ 4309.220168] Key type .llcrypt unregistered [ 4309.221748] Key type ._llcrypt unregistered [ 4336.235329] Key type ._llcrypt registered [ 4336.237412] Key type .llcrypt registered [ 4336.323768] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_hostid [ 4346.338903] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 4347.247034] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4347.340738] alg: No test for adler32 (adler32-zlib) [ 4348.346919] Lustre: Lustre: Build Version: 2.17.50_86_g2f7fa4d [ 4348.522152] LNet: Added LNI 192.168.201.107@tcp [8/256/0/180] [ 4350.168139] Key type lgssc registered [ 4350.966840] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4372.558686] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4372.584142] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4373.821254] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4373.843472] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4373.917595] Lustre: lustre-MDT0000: new disk, initializing [ 4373.971669] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4373.997368] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4377.016617] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 4383.476158] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4383.549057] Lustre: 140691:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4383.588098] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4383.591422] Lustre: Skipped 1 previous similar message [ 4383.645554] Lustre: lustre-MDT0001: new disk, initializing [ 4383.711413] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4383.735237] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4383.740357] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4386.306805] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 4391.118194] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4393.232416] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4397.634712] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4397.782594] Lustre: lustre-OST0000: new disk, initializing [ 4397.786091] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4397.843479] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4398.796355] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4398.801954] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4398.872731] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4401.614469] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 4406.285772] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4409.253205] Lustre: Failing over lustre-OST0000 [ 4409.260799] LustreError: 143121:0:(ldlm_resource.c:1170:ldlm_resource_complain()) filter-lustre-OST0000_UUID: namespace resource [0x280000401:0x2:0x0].0x0 (00000000dde96daa) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 4409.315066] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4409.315762] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4409.329795] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4409.332369] Lustre: Skipped 1 previous similar message [ 4409.339883] Lustre: Skipped 1 previous similar message [ 4411.323149] LustreError: 143121:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4411.450204] Lustre: server umount lustre-OST0000 complete [ 4426.946929] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4427.154809] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4427.163292] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4428.115085] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4429.235845] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 4429.236698] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4429.791608] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 4431.461818] Lustre: Failing over lustre-MDT0000 [ 4431.538825] LustreError: 144098:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4431.541217] LustreError: 144098:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4431.576505] Lustre: server umount lustre-MDT0000 complete [ 4434.402603] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4434.407607] Lustre: Skipped 2 previous similar messages [ 4438.323072] LustreError: 140699:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4438.330451] LustreError: 140699:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 4439.525062] LustreError: 140697:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4439.537443] LustreError: 140697:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4443.442905] LustreError: 140699:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4446.660928] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4446.755110] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4446.762932] Lustre: 140694:0:(mgc_request.c:1911:mgc_process_log()) MGC192.168.201.107@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 4446.912249] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4446.947225] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4448.567129] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4449.692600] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 4452.332593] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4452.335068] Lustre: Skipped 1 previous similar message [ 4452.345430] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 4452.372076] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4454.881710] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4454.887726] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4454.892775] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4457.446233] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4457.452261] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4457.457975] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4460.603270] LustreError: 144983:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4460.612361] LustreError: 144983:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4460.687838] Lustre: server umount lustre-OST0000 complete [ 4467.681753] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4467.683256] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4467.692359] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4467.700031] Lustre: Skipped 1 previous similar message [ 4467.708375] Lustre: Skipped 2 previous similar messages [ 4469.806378] LustreError: 145188:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 4469.814410] LustreError: 145188:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4469.944161] Lustre: server umount lustre-MDT0000 complete [ 4472.613948] LustreError: 141482:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769759370 with bad export cookie 790493042835849284 [ 4472.618288] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4472.620928] LustreError: 141482:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4472.823402] Lustre: server umount lustre-MDT0001 complete [ 4479.555825] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing unload_modules_local [ 4481.392639] Key type lgssc unregistered [ 4481.619192] LNet: 145920:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4481.625663] LNetError: 145920:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4481.633714] LNet: Removed LNI 192.168.201.107@tcp [ 4482.151334] Key type .llcrypt unregistered [ 4482.158318] Key type ._llcrypt unregistered [ 4510.268822] Key type ._llcrypt registered [ 4510.271072] Key type .llcrypt registered [ 4510.329132] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 4510.882860] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4510.903136] alg: No test for adler32 (adler32-zlib) [ 4511.841915] Lustre: Lustre: Build Version: 2.17.50_86_g2f7fa4d [ 4512.011912] LNet: Added LNI 192.168.201.107@tcp [8/256/0/180] [ 4513.624108] Key type lgssc registered [ 4514.325720] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4519.188890] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4519.202199] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4520.481405] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4523.341575] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 4528.497800] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4530.658242] LustreError: 147421:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4530.667032] LustreError: 147421:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4531.079507] LustreError: 147450:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4531.086933] LustreError: 147450:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4531.092200] LustreError: 147450:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 11, retries 0, failed: rc = -5 [ 4531.178977] LustreError: 147972:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4531.270865] Lustre: server umount lustre-MDT0000 complete [ 4539.982103] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_hostid [ 4545.486682] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 4566.548905] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4566.715650] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4566.734874] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4566.780442] Lustre: lustre-MDT0000: new disk, initializing [ 4566.832203] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4566.845687] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4569.450796] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 4576.501972] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4576.582851] Lustre: 150604:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4576.625482] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4576.627900] Lustre: Skipped 1 previous similar message [ 4576.675936] Lustre: lustre-MDT0001: new disk, initializing [ 4576.735500] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4576.772699] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4576.787156] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4579.612359] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 4584.705886] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4586.643982] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4590.500933] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4590.641472] Lustre: lustre-OST0000: new disk, initializing [ 4590.644089] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4590.684341] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4594.413376] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 4598.831699] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4598.837768] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4598.914614] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4599.925740] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4603.609461] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4604.577564] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 02:51:42 (1769759502) [ 4773.344879] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4773.346462] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4773.351176] Lustre: Skipped 2 previous similar messages [ 4773.356941] Lustre: Skipped 1 previous similar message [ 4778.465687] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4783.584820] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4783.591355] Lustre: Skipped 4 previous similar messages [ 4786.144121] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4786.199304] LustreError: 154111:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4786.201337] LustreError: 154111:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4786.239107] Lustre: server umount lustre-MDT0000 complete [ 4788.369726] LustreError: 150594:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769759686 with bad export cookie 17562658911837042477 [ 4788.372434] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4788.373705] LustreError: 150594:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4788.708240] LustreError: 150609:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4788.709808] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4788.724808] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4793.829979] LustreError: 150609:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4793.833252] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4794.513507] LustreError: 154313:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4794.518030] LustreError: 154313:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4794.606073] Lustre: server umount lustre-MDT0001 complete [ 4811.234169] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4811.278122] LustreError: 154514:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 4811.283076] LustreError: 154514:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4811.398906] Lustre: server umount lustre-OST0000 complete [ 4817.078293] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_hostid [ 4821.839204] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 4839.441712] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4839.587628] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4839.606947] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4839.647774] Lustre: lustre-MDT0000: new disk, initializing [ 4839.694791] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4839.711289] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4841.932777] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 4848.381360] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4848.451918] Lustre: 156843:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4848.492864] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4848.498990] Lustre: Skipped 1 previous similar message [ 4848.563589] Lustre: lustre-MDT0001: new disk, initializing [ 4848.607534] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4848.622641] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4848.630682] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4850.739544] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 4855.337755] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4857.282904] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4860.891406] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4861.052288] Lustre: lustre-OST0000: new disk, initializing [ 4861.056884] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4863.969483] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 4867.962550] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4870.102158] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4870.673616] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4870.679227] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4870.730106] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4871.256868] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 4873.642366] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4873.770588] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4875.748833] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4875.749364] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4875.756274] LustreError: Skipped 1 previous similar message [ 4875.758627] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4875.761535] Lustre: Skipped 1 previous similar message [ 4875.770742] Lustre: Skipped 1 previous similar message [ 4876.926406] LustreError: 159780:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4876.931178] LustreError: 159780:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4877.008849] Lustre: server umount lustre-OST0000 complete [ 4879.328852] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4879.336712] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4885.113144] LustreError: 159981:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4885.118357] LustreError: 159981:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4885.196379] Lustre: server umount lustre-MDT0000 complete [ 4885.985975] LustreError: 157681:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4885.998122] LustreError: 157681:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4887.495347] LustreError: 159982:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769759785 with bad export cookie 17562658911837201307 [ 4887.500978] LustreError: 159982:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4887.502292] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4887.685107] Lustre: server umount lustre-MDT0001 complete [ 4892.509493] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 02:56:30 (1769759790) [ 4899.986101] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_hostid [ 4904.543542] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 4920.877035] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4920.998310] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4921.018467] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4921.059341] Lustre: lustre-MDT0000: new disk, initializing [ 4921.096729] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4921.099697] Lustre: Skipped 1 previous similar message [ 4921.108774] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4922.778068] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 4928.538478] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4928.598753] Lustre: 163157:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4928.748608] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4930.858211] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 4934.862886] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4936.501855] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4939.497603] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4942.259653] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 4944.387885] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4944.439073] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4946.314478] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4949.475471] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4949.476296] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4949.485301] Lustre: Skipped 1 previous similar message [ 4949.487597] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4949.489606] Lustre: Skipped 4 previous similar messages [ 4955.581106] LustreError: 165693:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4955.586343] LustreError: 165693:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 4955.683717] Lustre: server umount lustre-OST0000 complete [ 4959.716496] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4959.718062] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4959.727756] Lustre: Skipped 1 previous similar message [ 4959.733367] LustreError: Skipped 1 previous similar message [ 4964.005859] Lustre: server umount lustre-MDT0000 complete [ 4964.833928] LustreError: 164843:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4964.846255] LustreError: 164843:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4966.162604] LustreError: 163147:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769759864 with bad export cookie 17562658911837202175 [ 4966.165760] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4966.173521] LustreError: 163147:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4966.339876] Lustre: server umount lustre-MDT0001 complete [ 4970.928949] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 02:57:48 (1769759868) [ 4978.772977] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_hostid [ 4983.503048] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 5002.032833] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5002.130867] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5002.144494] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5002.147345] Lustre: Skipped 3 previous similar messages [ 5002.195091] Lustre: lustre-MDT0000: new disk, initializing [ 5002.196752] Lustre: Skipped 2 previous similar messages [ 5002.237443] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5002.244357] Lustre: Skipped 2 previous similar messages [ 5002.257232] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5002.262241] Lustre: Skipped 2 previous similar messages [ 5003.899199] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 5009.428293] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5009.505982] Lustre: 169071:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5009.637107] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5011.507906] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 5015.859576] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5017.809888] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5021.491162] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5024.878916] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 5027.420944] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5029.247757] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5037.539321] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5037.545577] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5037.552294] Lustre: Skipped 1 previous similar message [ 5037.557262] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5037.559517] Lustre: Skipped 5 previous similar messages [ 5038.655083] LustreError: 171608:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5038.659210] LustreError: 171608:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 5038.746039] Lustre: server umount lustre-OST0000 complete [ 5047.017094] Lustre: server umount lustre-MDT0000 complete [ 5047.777674] LustreError: 169861:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5047.785083] LustreError: 169861:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5049.171727] LustreError: 169063:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769759947 with bad export cookie 17562658911837203260 [ 5049.176200] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5049.178196] LustreError: 169063:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5049.325026] Lustre: server umount lustre-MDT0001 complete [ 5055.442457] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing unload_modules_local [ 5056.972793] Key type lgssc unregistered [ 5057.125727] LNet: 172543:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5057.132408] LNetError: 172543:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5057.141660] LNet: Removed LNI 192.168.201.107@tcp [ 5057.579214] Key type .llcrypt unregistered [ 5057.586537] Key type ._llcrypt unregistered [ 5071.632473] Key type ._llcrypt registered [ 5071.635232] Key type .llcrypt registered [ 5071.709378] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 5072.211193] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5072.259546] alg: No test for adler32 (adler32-zlib) [ 5073.190390] Lustre: Lustre: Build Version: 2.17.50_86_g2f7fa4d [ 5073.324097] LNet: Added LNI 192.168.201.107@tcp [8/256/0/180] [ 5074.960131] Key type lgssc registered [ 5075.561216] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5079.928164] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5079.938373] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5081.179928] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5083.376621] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 5088.810240] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5089.036213] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5091.534845] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 5094.633829] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5096.836669] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5100.468035] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5100.674819] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5103.603370] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 5106.121322] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5106.853870] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5129.704472] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5129.710592] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5129.713142] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5129.718165] Lustre: Skipped 1 previous similar message [ 5134.819344] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5134.822804] Lustre: Skipped 1 previous similar message [ 5134.840166] LustreError: 176181:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5134.906092] Lustre: server umount lustre-OST0000 complete [ 5139.949201] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5139.952757] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5139.954451] Lustre: Skipped 2 previous similar messages [ 5139.966978] Lustre: Skipped 1 previous similar message [ 5145.057325] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5145.062783] Lustre: Skipped 1 previous similar message [ 5150.181283] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5150.188031] Lustre: Skipped 1 previous similar message [ 5151.200828] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5151.267579] LustreError: 176384:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5151.273928] LustreError: 176384:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5151.330954] Lustre: server umount lustre-MDT0000 complete [ 5153.356530] LustreError: 176182:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769760051 with bad export cookie 5071754940722162814 [ 5153.358424] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5153.363561] LustreError: 176182:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5153.455663] LustreError: 176587:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5153.459682] LustreError: 176587:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5153.580140] Lustre: server umount lustre-MDT0001 complete [ 5159.480626] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing unload_modules_local [ 5160.991686] Key type lgssc unregistered [ 5161.152635] LNet: 177118:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5161.155939] LNetError: 177118:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5161.166346] LNet: Removed LNI 192.168.201.107@tcp [ 5161.628095] Key type .llcrypt unregistered [ 5161.629397] Key type ._llcrypt unregistered [ 5174.427893] Key type ._llcrypt registered [ 5174.429467] Key type .llcrypt registered [ 5174.482921] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 5175.045789] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5175.060514] alg: No test for adler32 (adler32-zlib) [ 5175.974251] Lustre: Lustre: Build Version: 2.17.50_86_g2f7fa4d [ 5176.098092] LNet: Added LNI 192.168.201.107@tcp [8/256/0/180] [ 5177.704157] Key type lgssc registered [ 5178.359657] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5182.650315] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5182.659940] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5183.932414] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5185.855953] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 5190.686273] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5190.927975] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5193.575660] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 5196.744527] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5198.826592] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5202.070992] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5202.246432] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5205.420742] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 5207.958449] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5209.450160] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5221.858973] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5221.859421] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5221.863342] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5221.870696] Lustre: Skipped 1 previous similar message [ 5223.159072] LustreError: 180767:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5223.213647] Lustre: server umount lustre-OST0000 complete [ 5225.745839] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff9a24c6840800.ost_server_uuid 50 [ 5242.347792] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5257.697824] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5257.698026] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5257.700612] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5257.700620] Lustre: Skipped 1 previous similar message [ 5257.725227] Lustre: Skipped 1 previous similar message [ 5258.829463] LustreError: 181137:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5258.831972] LustreError: 181137:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5258.890458] Lustre: server umount lustre-MDT0000 complete [ 5260.743276] LustreError: 180119:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769760158 with bad export cookie 8051806584625741608 [ 5260.747938] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5260.824797] LustreError: 181338:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5260.826631] LustreError: 181338:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5260.898909] Lustre: server umount lustre-MDT0001 complete [ 5264.672714] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 03:02:42 (1769760162) [ 5267.554475] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5267.767065] LustreError: 182008:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5267.775046] LustreError: 182008:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5267.801248] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5269.259019] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 5272.888890] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5273.083203] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5274.708734] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 5276.859229] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5278.313498] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5280.941299] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5281.096433] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5283.459320] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 5285.600665] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5288.348598] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5288.429247] Lustre: lustre-OST0001: new disk, initializing [ 5288.431820] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5290.642894] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 5292.585291] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5294.209879] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5294.607531] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5294.611740] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5294.650042] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5297.633589] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5297.636152] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5297.641152] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5297.643607] Lustre: Skipped 1 previous similar message [ 5299.685288] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5299.687539] Lustre: Skipped 3 previous similar messages [ 5301.359136] LustreError: 185204:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5301.361372] LustreError: 185204:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5301.378810] Lustre: server umount lustre-OST0000 complete [ 5303.547395] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5303.632939] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5304.630711] LustreError: 183600:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5305.301832] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5305.393282] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5309.743483] LustreError: 183600:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5309.752722] LustreError: 183600:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5315.043204] LustreError: 183885:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5315.050616] LustreError: 183885:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5320.162071] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5320.162906] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5320.166560] LustreError: 183885:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5320.166587] LustreError: Skipped 2 previous similar messages [ 5320.171962] Lustre: Skipped 1 previous similar message [ 5320.178889] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5320.181271] Lustre: Skipped 1 previous similar message [ 5321.721169] LustreError: 185904:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 5321.724401] LustreError: 185904:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5321.768546] Lustre: server umount lustre-OST0001 complete [ 5324.257723] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5324.261699] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5329.459971] Lustre: server umount lustre-MDT0000 complete [ 5330.401830] LustreError: 182007:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5330.408822] LustreError: 182007:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5331.084111] LustreError: 181988:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769760229 with bad export cookie 8051806584625742665 [ 5331.087210] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5331.088219] LustreError: 181988:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 5331.161645] LustreError: 186308:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 5331.163548] LustreError: 186308:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 5331.242517] Lustre: server umount lustre-MDT0001 complete [ 5341.454733] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 03:03:59 (1769760239) [ 5344.298979] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5344.450441] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5344.467838] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5344.569252] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5344.573477] Lustre: Skipped 1 previous similar message [ 5346.157297] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 5349.919699] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5349.968031] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5349.983058] Lustre: 187774:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5351.911850] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 5354.139751] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5355.827972] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5358.716212] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5358.797704] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5358.840854] LustreError: 189398:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5361.045916] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 5363.530817] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5364.202054] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 5366.486366] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5366.525290] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5368.843692] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 5371.208223] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5373.947754] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5373.950118] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5373.953938] Lustre: Skipped 2 previous similar messages [ 5373.955914] Lustre: Skipped 3 previous similar messages [ 5378.480131] LustreError: 190953:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5378.482173] LustreError: 190953:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 5378.503629] Lustre: server umount lustre-OST0000 complete [ 5394.405508] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5394.407381] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5394.410041] LustreError: Skipped 1 previous similar message [ 5394.410166] LustreError: 190017:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5394.410171] LustreError: 190017:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 5394.416754] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5394.430324] Lustre: Skipped 3 previous similar messages [ 5395.606676] Lustre: server umount lustre-OST0001 complete [ 5403.372433] Lustre: server umount lustre-MDT0000 complete [ 5405.192193] LustreError: 187766:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769760303 with bad export cookie 8051806584625744051 [ 5405.196354] LustreError: 187766:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5405.197268] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5405.339345] Lustre: server umount lustre-MDT0001 complete [ 5414.768721] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 03:05:12 (1769760312) [ 5421.016282] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_hostid [ 5425.247779] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 5441.328129] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5441.494208] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5441.516723] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5441.556956] Lustre: lustre-MDT0000: new disk, initializing [ 5441.597515] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5441.601297] Lustre: Skipped 3 previous similar messages [ 5441.610954] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5443.447181] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 5450.465782] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5450.554064] Lustre: 195355:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5450.593060] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5450.598192] Lustre: Skipped 1 previous similar message [ 5450.695548] Lustre: lustre-MDT0001: new disk, initializing [ 5450.845818] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5450.863868] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5455.095863] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 5462.945193] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5466.771261] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5474.405343] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5474.620518] Lustre: lustre-OST0000: new disk, initializing [ 5474.623474] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5478.559403] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 5481.548647] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5481.553665] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5481.644693] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5483.737865] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5485.914350] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5486.008666] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5487.942254] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5488.038610] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5492.196126] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5492.196158] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5492.199376] LustreError: Skipped 1 previous similar message [ 5492.205086] Lustre: Skipped 3 previous similar messages [ 5492.206220] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5492.214310] Lustre: Skipped 3 previous similar messages [ 5495.149767] LustreError: 198285:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5495.152015] LustreError: 198285:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 5495.199069] Lustre: server umount lustre-OST0000 complete [ 5503.296922] Lustre: server umount lustre-MDT0000 complete [ 5505.284426] LustreError: 195348:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769760403 with bad export cookie 8051806584625745570 [ 5505.285297] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5505.291377] LustreError: 195348:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5505.450841] Lustre: server umount lustre-MDT0001 complete [ 5509.670329] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5509.889835] LustreError: 199165:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5509.896918] LustreError: 199165:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5509.920967] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5509.925165] Lustre: Skipped 2 previous similar messages [ 5511.862812] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 5516.532485] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5518.575407] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 5521.273633] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5523.394105] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5527.254345] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5530.689215] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 5533.426463] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5535.738119] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5535.849470] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5538.203208] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5542.471536] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 4 sec [ 5545.953550] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5549.603873] Lustre: server umount lustre-OST0000 complete [ 5552.522335] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5554.311895] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5579.179966] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5581.638617] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 5584.086802] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5584.733129] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5585.991911] LustreError: 203563:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5585.994527] LustreError: 203563:0:(obd_class.h:479:obd_check_dev()) Skipped 21 previous similar messages [ 5586.039195] Lustre: server umount lustre-OST0000 complete [ 5587.768016] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5587.770327] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5587.775864] Lustre: Skipped 6 previous similar messages [ 5587.777995] Lustre: Skipped 7 previous similar messages [ 5593.944196] Lustre: server umount lustre-MDT0000 complete [ 5595.551609] LustreError: 199144:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769760493 with bad export cookie 8051806584625746438 [ 5595.553934] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5595.559427] LustreError: 199144:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5595.695172] Lustre: server umount lustre-MDT0001 complete [ 5599.604603] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 03:08:17 (1769760497) [ 5602.592585] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5604.473736] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 5608.216080] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5610.135052] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 5612.757748] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5614.263328] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5617.416113] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5620.439815] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 5622.725136] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5624.575574] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5627.737762] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 5629.431904] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5629.526642] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5632.411832] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5632.486294] Lustre: lustre-OST0001: new disk, initializing [ 5632.488993] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5634.990742] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 5639.000446] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5640.729839] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5642.774220] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5642.779081] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5642.830372] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5643.907583] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 5645.490158] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5645.583308] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5647.841695] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5647.844702] LustreError: Skipped 1 previous similar message [ 5652.431126] Lustre: server umount lustre-OST0001 complete [ 5652.965544] LustreError: 206232:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5652.970049] LustreError: 206232:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 5654.675031] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5654.759371] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5656.305590] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5656.389934] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5680.260641] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5680.346060] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5680.348863] Lustre: Skipped 7 previous similar messages [ 5682.360889] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 5684.406038] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5685.039739] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5685.735471] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 5685.742149] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 5685.742284] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5685.750035] Lustre: Skipped 1 previous similar message [ 5687.937885] Lustre: server umount lustre-OST0001 complete [ 5706.322763] Lustre: server umount lustre-OST0000 complete [ 5713.886756] Lustre: server umount lustre-MDT0000 complete [ 5715.320833] LustreError: 204619:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769760613 with bad export cookie 8051806584625747369 [ 5715.323921] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5715.325116] LustreError: 204619:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5721.470843] LustreError: 211153:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 5721.473468] LustreError: 211153:0:(obd_class.h:479:obd_check_dev()) Skipped 37 previous similar messages [ 5721.534200] Lustre: server umount lustre-MDT0001 complete [ 5731.162501] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 03:10:29 (1769760629) [ 5733.708756] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5733.838210] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5733.852503] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5735.508671] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 5739.019745] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5739.084704] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5739.098239] Lustre: 212621:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5740.713263] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 5742.671852] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5744.044903] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5746.528727] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5746.603628] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5749.037440] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 5751.481878] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5761.458219] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5761.502932] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5764.300525] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 5766.940530] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5767.641152] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5769.308494] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5770.412441] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 5771.889565] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5771.973957] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5773.438684] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9a24d0f0e000.ost_server_uuid 50 [ 5774.019385] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9a24d0f0e000.ost_server_uuid in FULL state after 0 sec [ 5774.417903] Lustre: Permanently deactivating lustre-OST0001 [ 5774.423037] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 5775.462315] Lustre: Permanently reactivating lustre-OST0001 [ 5779.941315] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5779.942963] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5779.945204] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5779.945211] Lustre: Skipped 13 previous similar messages [ 5779.946893] Lustre: Skipped 9 previous similar messages [ 5779.950137] LustreError: Skipped 6 previous similar messages [ 5782.789759] Lustre: server umount lustre-OST0001 complete [ 5798.368295] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5798.433960] Lustre: server umount lustre-OST0000 complete [ 5814.240124] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5814.318489] Lustre: server umount lustre-MDT0000 complete [ 5816.002212] LustreError: 212612:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769760714 with bad export cookie 8051806584625748643 [ 5816.008062] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5816.140776] Lustre: server umount lustre-MDT0001 complete [ 5826.680984] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 03:12:04 (1769760724) [ 5830.121246] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5830.235110] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5830.247869] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5830.253441] Lustre: Skipped 5 previous similar messages [ 5832.041178] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 5835.699160] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5835.751360] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5835.762469] Lustre: 218735:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5837.455357] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 5839.831641] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5841.627608] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5844.637578] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5844.732263] Lustre: Permanently deactivating lustre-OST0000 [ 5844.740841] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 5844.991829] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5847.150373] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 5849.250332] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5851.835092] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5854.110173] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 5854.768559] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 5856.278199] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5856.949310] Lustre: Permanently reactivating lustre-OST0000 [ 5861.889265] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 5861.892674] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5861.897461] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5861.911868] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 5861.920288] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 5875.589853] Lustre: server umount lustre-OST0001 complete [ 5883.228681] Lustre: server umount lustre-OST0000 complete [ 5899.232118] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5899.302127] Lustre: server umount lustre-MDT0000 complete [ 5900.868220] LustreError: 218770:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769760799 with bad export cookie 8051806584625750309 [ 5900.871214] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5900.873020] LustreError: 218770:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 5901.008105] Lustre: server umount lustre-MDT0001 complete [ 5905.340283] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing unload_modules_local [ 5906.581111] Key type lgssc unregistered [ 5906.750460] LNet: 223141:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5906.753992] LNetError: 223141:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5906.764391] LNet: Removed LNI 192.168.201.107@tcp [ 5907.096713] Key type .llcrypt unregistered [ 5907.097945] Key type ._llcrypt unregistered [ 5916.890300] Key type ._llcrypt registered [ 5916.891508] Key type .llcrypt registered [ 5916.935000] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 5917.285427] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5917.334277] alg: No test for adler32 (adler32-zlib) [ 5918.216760] Lustre: Lustre: Build Version: 2.17.50_86_g2f7fa4d [ 5918.310807] LNet: Added LNI 192.168.201.107@tcp [8/256/0/180] [ 5919.896172] Key type lgssc registered [ 5920.307567] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5924.103475] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5924.111599] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5925.340677] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5927.032461] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 5930.882802] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5930.982820] Lustre: Found index 1 for lustre-MDT0001, updating log [ 5931.001969] Lustre: 224552:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5931.007067] Lustre: Permanently deactivating lustre-MDT0001 [ 5931.013984] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 5931.150173] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5931.279099] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 5931.281877] LustreError: 224595:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 5931.285868] LustreError: 224595:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 5931.289557] LustreError: 224595:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 6, retries 1, failed: rc = -108 [ 5932.805563] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 5935.191988] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5936.742441] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5939.638401] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5939.778652] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5941.860779] LustreError: 226511:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5941.867194] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 5941.874657] LustreError: 226511:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5941.890429] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 5944.238932] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5947.052304] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5947.130418] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5947.892932] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 5949.352818] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 5949.929461] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 5949.931068] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 5951.742497] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5953.495986] Lustre: Permanently reactivating lustre-MDT0001 [ 5953.504521] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 5953.509259] Lustre: Skipped 2 previous similar messages [ 5962.041242] Lustre: lustre-MDT0001: Client df70d3dc-cfa1-46da-88b8-d9fef161651c (at 192.168.201.7@tcp) reconnecting [ 5964.230916] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5964.834075] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5966.620818] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 5966.706266] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 5968.252399] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 5968.330902] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 5969.245760] Lustre: Permanently deactivating lustre-MDT0001 [ 5969.252683] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 5969.254946] Lustre: Skipped 2 previous similar messages [ 5977.095334] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 5985.763697] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5985.767046] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5985.769211] Lustre: Skipped 2 previous similar messages [ 5985.771996] Lustre: Skipped 1 previous similar message [ 5989.894719] LustreError: 228719:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 5989.922582] Lustre: server umount lustre-MDT0000 complete [ 5990.882310] LustreError: 225291:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5990.888927] LustreError: 225291:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5991.558615] LustreError: 225285:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769760889 with bad export cookie 1659951632328416443 [ 5991.561983] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5991.564378] LustreError: 225285:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5991.630777] LustreError: 228920:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 5991.633551] LustreError: 228920:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5991.695332] Lustre: server umount lustre-MDT0001 complete [ 5999.278100] LustreError: 229120:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 5999.280984] LustreError: 229120:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5999.298876] Lustre: server umount lustre-OST0000 complete [ 6003.574433] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 03:15:01 (1769760901) [ 6007.517263] LustreError: 230124:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 6007.519467] LustreError: 230124:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 6007.567475] Lustre: server umount lustre-OST0001 complete [ 6010.006724] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_hostid [ 6013.688573] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 6026.831882] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6026.920396] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6026.924826] Lustre: Skipped 2 previous similar messages [ 6026.937575] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6026.976255] Lustre: lustre-MDT0000: new disk, initializing [ 6027.010661] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6027.019329] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6028.448993] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 6033.004612] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6033.049799] Lustre: 232295:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6033.080029] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6033.082842] Lustre: Skipped 1 previous similar message [ 6033.119948] Lustre: lustre-MDT0001: new disk, initializing [ 6033.170362] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6033.174506] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6034.598807] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 6038.024661] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6039.480417] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6042.114972] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6042.225058] Lustre: lustre-OST0000: new disk, initializing [ 6042.227231] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6042.260400] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6042.264488] Lustre: Skipped 1 previous similar message [ 6042.501241] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6042.506306] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6042.542552] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6044.418379] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 6047.856507] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6048.698363] LustreError: 232301:0:(mdt_handler.c:3252:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 6053.231903] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6053.295538] Lustre: lustre-OST0001: new disk, initializing [ 6053.297762] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6055.437352] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 6058.637101] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6058.792149] LustreError: 232301:0:(mdt_handler.c:3252:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 6058.800554] LustreError: 232299:0:(mdt_handler.c:3252:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 6059.018424] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6059.022442] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6059.055626] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6059.227263] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6068.888122] LustreError: 232299:0:(mdt_handler.c:3252:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 6070.071896] Lustre: lustre-OST0001: Not available for connect from 192.168.201.7@tcp (stopping) [ 6074.338373] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6074.341089] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6074.342284] LustreError: Skipped 1 previous similar message [ 6074.346530] Lustre: Skipped 1 previous similar message [ 6074.347398] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6074.352415] Lustre: Skipped 1 previous similar message [ 6075.700462] LustreError: 235778:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 6075.703922] LustreError: 235778:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6075.727343] Lustre: server umount lustre-OST0001 complete [ 6079.460726] LustreError: 234086:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6079.466118] LustreError: 234086:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6084.581361] LustreError: 234020:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6084.590818] LustreError: 234020:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6088.673348] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6088.673612] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6088.681559] LustreError: Skipped 1 previous similar message [ 6088.685820] Lustre: Skipped 1 previous similar message [ 6088.688623] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6088.691915] Lustre: Skipped 2 previous similar messages [ 6089.703387] LustreError: 234020:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6089.709462] LustreError: 234020:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6094.067081] LustreError: 236000:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6094.069648] LustreError: 236000:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6094.117304] Lustre: server umount lustre-OST0000 complete [ 6099.937553] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6099.937657] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6099.945416] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6099.947682] Lustre: Skipped 3 previous similar messages [ 6101.789297] Lustre: server umount lustre-MDT0000 complete [ 6103.275273] LustreError: 234035:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769761001 with bad export cookie 1659951632328419222 [ 6103.280172] LustreError: 234035:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6103.280276] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6109.493371] Lustre: server umount lustre-MDT0001 complete [ 6113.381755] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing unload_modules_local [ 6114.523494] Key type lgssc unregistered [ 6114.659508] LNet: 236935:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6114.662640] LNetError: 236935:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6114.670347] LNet: Removed LNI 192.168.201.107@tcp [ 6114.992123] Key type .llcrypt unregistered [ 6114.993304] Key type ._llcrypt unregistered [ 6130.211696] Key type ._llcrypt registered [ 6130.212760] Key type .llcrypt registered [ 6130.252474] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 6130.697831] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6130.716041] alg: No test for adler32 (adler32-zlib) [ 6131.570849] Lustre: Lustre: Build Version: 2.17.50_86_g2f7fa4d [ 6131.658906] LNet: Added LNI 192.168.201.107@tcp [8/256/0/180] [ 6133.240146] Key type lgssc registered [ 6133.621180] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6136.376821] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6136.383252] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6137.488874] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6137.497282] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6137.560066] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6138.939124] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 6141.968544] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6142.006368] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6142.016315] Lustre: 239098:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6142.105671] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6143.485286] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 6145.366116] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6146.590052] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6148.888396] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6148.949301] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6149.021243] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6150.913690] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 6152.843504] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6154.767266] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 6200.803655] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6200.803933] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6200.804818] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6200.809193] Lustre: Skipped 1 previous similar message [ 6200.812041] LustreError: Skipped 1 previous similar message [ 6202.801081] LustreError: 241453:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6202.848601] Lustre: server umount lustre-OST0000 complete [ 6204.447741] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 6208.793027] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6208.911456] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6210.974504] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 6212.793694] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6221.157450] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6221.162262] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6221.162266] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6221.169418] Lustre: Skipped 1 previous similar message [ 6246.881970] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6246.883599] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6246.884823] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6246.884827] Lustre: Skipped 1 previous similar message [ 6246.885192] LustreError: Skipped 1 previous similar message [ 6246.889195] Lustre: Skipped 1 previous similar message [ 6252.002423] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6252.004198] Lustre: Skipped 2 previous similar messages [ 6252.460060] LustreError: 243116:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6252.462642] LustreError: 243116:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6252.514789] Lustre: server umount lustre-OST0000 complete [ 6255.009823] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6255.010927] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6255.014072] Lustre: Skipped 1 previous similar message [ 6255.019900] Lustre: Skipped 1 previous similar message [ 6260.193134] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6260.197097] Lustre: Skipped 1 previous similar message [ 6267.872141] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6267.914244] LustreError: 243317:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6267.916868] LustreError: 243317:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6267.952657] Lustre: server umount lustre-MDT0000 complete [ 6269.291216] LustreError: 243016:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769761167 with bad export cookie 6177499857084744447 [ 6269.293933] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6269.295571] LustreError: 243016:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6269.409731] Lustre: server umount lustre-MDT0001 complete [ 6273.399398] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing unload_modules_local [ 6274.465703] Key type lgssc unregistered [ 6274.593411] LNet: 244050:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6274.597658] LNetError: 244050:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6274.607331] LNet: Removed LNI 192.168.201.107@tcp [ 6274.944701] Key type .llcrypt unregistered [ 6274.945849] Key type ._llcrypt unregistered [ 6284.574778] Key type ._llcrypt registered [ 6284.576067] Key type .llcrypt registered [ 6284.613264] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 6285.027385] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6285.048667] alg: No test for adler32 (adler32-zlib) [ 6285.915756] Lustre: Lustre: Build Version: 2.17.50_86_g2f7fa4d [ 6285.999165] LNet: Added LNI 192.168.201.107@tcp [8/256/0/180] [ 6287.584260] Key type lgssc registered [ 6287.979876] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6290.730839] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6290.738601] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6291.907785] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6293.231681] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 6296.469974] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6296.612237] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6297.965919] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 6299.902057] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6301.281117] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6303.704499] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6303.824093] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6305.801771] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 6307.812294] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6317.162349] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 6326.753137] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6326.756566] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6326.762516] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6327.777806] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6327.778023] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6327.778781] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6332.020104] LustreError: 248456:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6332.063538] Lustre: server umount lustre-OST0000 complete [ 6337.504687] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6337.506788] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6337.510385] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6337.512039] Lustre: Skipped 1 previous similar message [ 6339.449945] LustreError: 248659:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6339.451875] LustreError: 248659:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6339.477589] Lustre: server umount lustre-MDT0000 complete [ 6340.696718] LustreError: 245402:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769761238 with bad export cookie 16707207513689519160 [ 6340.698409] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6340.701662] LustreError: 245402:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6340.762538] LustreError: 248860:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6340.764129] LustreError: 248860:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6340.816302] Lustre: server umount lustre-MDT0001 complete [ 6344.337105] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing unload_modules_local [ 6345.303763] Key type lgssc unregistered [ 6345.443395] LNet: 249391:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6345.445614] LNetError: 249391:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6345.458277] LNet: Removed LNI 192.168.201.107@tcp [ 6345.742172] Key type .llcrypt unregistered [ 6345.743034] Key type ._llcrypt unregistered [ 6352.530380] Key type ._llcrypt registered [ 6352.531406] Key type .llcrypt registered [ 6352.566629] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 6352.862823] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6352.872869] alg: No test for adler32 (adler32-zlib) [ 6353.728401] Lustre: Lustre: Build Version: 2.17.50_86_g2f7fa4d [ 6353.806840] LNet: Added LNI 192.168.201.107@tcp [8/256/0/180] [ 6355.384169] Key type lgssc registered [ 6355.729798] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6358.169082] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6358.175103] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6359.315376] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6360.634782] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 6363.505290] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6363.617365] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6364.849511] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 6366.610808] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6367.759391] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6369.898514] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6370.008296] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6371.760294] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 6373.465392] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6384.360176] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6393.824982] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6393.828463] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6393.833476] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6394.851342] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6394.852543] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6394.854108] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6398.637125] LustreError: 252977:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6398.673908] Lustre: server umount lustre-OST0000 complete [ 6404.576837] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6404.579735] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6404.584245] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6404.586964] Lustre: Skipped 1 previous similar message [ 6410.209073] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6410.211424] Lustre: Skipped 3 previous similar messages [ 6414.304111] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6414.334254] LustreError: 253182:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6414.337135] LustreError: 253182:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6414.363282] Lustre: server umount lustre-MDT0000 complete [ 6415.329721] LustreError: 250568:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6415.336588] LustreError: 250568:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6415.666183] LustreError: 250552:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769761313 with bad export cookie 10611084195839568054 [ 6415.669657] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6415.670581] LustreError: 250552:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6415.733258] LustreError: 253384:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6415.735153] LustreError: 253384:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6415.780347] Lustre: server umount lustre-MDT0001 complete [ 6419.475878] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing unload_modules_local [ 6420.572804] Key type lgssc unregistered [ 6420.716659] LNet: 253915:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6420.719975] LNetError: 253915:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6420.727413] LNet: Removed LNI 192.168.201.107@tcp [ 6421.045367] Key type .llcrypt unregistered [ 6421.046972] Key type ._llcrypt unregistered [ 6429.914518] Key type ._llcrypt registered [ 6429.915438] Key type .llcrypt registered [ 6429.945646] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 6430.223099] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6430.254452] alg: No test for adler32 (adler32-zlib) [ 6431.108440] Lustre: Lustre: Build Version: 2.17.50_86_g2f7fa4d [ 6431.190542] LNet: Added LNI 192.168.201.107@tcp [8/256/0/180] [ 6432.768155] Key type lgssc registered [ 6433.111510] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6435.578392] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6435.585029] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6436.722948] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6438.021983] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 6440.774886] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6440.897044] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6442.159952] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 6443.911707] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6445.191540] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6447.324551] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6447.429614] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6449.213383] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 6451.020236] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6461.798412] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6474.722502] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 6474.725062] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6474.729681] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6477.282691] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6477.283019] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6477.287402] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6480.047177] LustreError: 258323:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6480.088058] Lustre: server umount lustre-OST0000 complete [ 6481.888910] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6481.892244] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6481.897686] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6481.901211] Lustre: Skipped 1 previous similar message [ 6487.521139] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6487.523382] Lustre: Skipped 3 previous similar messages [ 6487.550163] LustreError: 258524:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6487.552626] LustreError: 258524:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6487.574862] Lustre: server umount lustre-MDT0000 complete [ 6488.783290] LustreError: 255268:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769761386 with bad export cookie 3208929549813426657 [ 6488.783782] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6488.786580] LustreError: 255268:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6488.850347] LustreError: 258727:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6488.852178] LustreError: 258727:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6488.901822] Lustre: server umount lustre-MDT0001 complete [ 6492.325506] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing unload_modules_local [ 6493.276346] Key type lgssc unregistered [ 6493.395470] LNet: 259259:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6493.398324] LNetError: 259259:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6493.407329] LNet: Removed LNI 192.168.201.107@tcp [ 6493.698022] Key type .llcrypt unregistered [ 6493.699174] Key type ._llcrypt unregistered [ 6500.455180] Key type ._llcrypt registered [ 6500.456098] Key type .llcrypt registered [ 6500.487848] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 6500.774333] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6500.781876] alg: No test for adler32 (adler32-zlib) [ 6501.629260] Lustre: Lustre: Build Version: 2.17.50_86_g2f7fa4d [ 6501.704319] LNet: Added LNI 192.168.201.107@tcp [8/256/0/180] [ 6503.288156] Key type lgssc registered [ 6503.615018] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6505.912543] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6505.917940] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6507.071403] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6508.216245] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 6510.794883] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6510.907248] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6512.086353] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 6513.796208] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6514.925457] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6517.004455] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6517.108274] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6518.120941] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 6518.870236] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 6520.578239] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6526.436465] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6526.437752] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6526.440815] Lustre: Skipped 1 previous similar message [ 6526.444713] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6530.990714] LustreError: 262862:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6531.024563] Lustre: server umount lustre-OST0000 complete [ 6536.673590] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6536.673735] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6536.674250] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6536.674254] Lustre: Skipped 1 previous similar message [ 6536.680647] Lustre: Skipped 1 previous similar message [ 6538.300142] LustreError: 263063:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6538.303489] LustreError: 263063:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6538.330926] Lustre: server umount lustre-MDT0000 complete [ 6539.576106] LustreError: 262093:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769761437 with bad export cookie 3287131096177087849 [ 6539.579356] LustreError: 262093:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6539.580486] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6539.640905] LustreError: 263264:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6539.642485] LustreError: 263264:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6539.693882] Lustre: server umount lustre-MDT0001 complete [ 6543.098251] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing unload_modules_local [ 6544.085839] Key type lgssc unregistered [ 6544.213366] LNet: 263796:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6544.217522] LNetError: 263796:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6544.228284] LNet: Removed LNI 192.168.201.107@tcp [ 6544.516560] Key type .llcrypt unregistered [ 6544.517872] Key type ._llcrypt unregistered [ 6547.758604] Key type ._llcrypt registered [ 6547.759628] Key type .llcrypt registered [ 6547.797035] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 6552.993055] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_hostid [ 6558.743265] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 6559.021256] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6559.027290] alg: No test for adler32 (adler32-zlib) [ 6559.869574] Lustre: Lustre: Build Version: 2.17.50_86_g2f7fa4d [ 6559.944768] LNet: Added LNI 192.168.201.107@tcp [8/256/0/180] [ 6561.520154] Key type lgssc registered [ 6561.853414] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6572.711576] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6572.717814] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6573.812187] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6573.822677] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6573.855571] Lustre: lustre-MDT0000: new disk, initializing [ 6573.879711] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6573.886687] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6575.165016] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 6579.355684] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6579.396734] Lustre: 267266:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6579.420875] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6579.423024] Lustre: Skipped 1 previous similar message [ 6579.457814] Lustre: lustre-MDT0001: new disk, initializing [ 6579.488368] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6579.500451] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6579.504970] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6580.766914] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 6583.763179] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6584.929632] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6587.020698] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6587.118088] Lustre: lustre-OST0000: new disk, initializing [ 6587.120729] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6587.147483] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6588.965188] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 6590.328109] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6590.331066] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6590.361415] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6591.933710] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6593.239318] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6593.312771] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6594.654871] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6594.736391] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6595.553539] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6595.555303] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6595.556082] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6595.563220] Lustre: Skipped 1 previous similar message [ 6600.674486] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6600.677116] Lustre: Skipped 2 previous similar messages [ 6601.454134] LustreError: 270194:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6601.494565] Lustre: server umount lustre-OST0000 complete [ 6605.280982] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6605.284094] LustreError: Skipped 1 previous similar message [ 6605.285952] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6605.292018] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6610.913155] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6610.915695] Lustre: Skipped 2 previous similar messages [ 6616.033381] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6616.036068] Lustre: Skipped 1 previous similar message [ 6617.056088] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6617.092870] LustreError: 270396:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6617.095258] LustreError: 270396:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6617.124296] Lustre: server umount lustre-MDT0000 complete [ 6618.330405] LustreError: 267258:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769761516 with bad export cookie 3513108834179618001 [ 6618.333244] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6618.334324] LustreError: 267258:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6618.400312] LustreError: 270599:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6618.402960] LustreError: 270599:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6618.454495] Lustre: server umount lustre-MDT0001 complete [ 6621.248268] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 03:25:19 (1769761519) [ 6623.461977] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6623.643950] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6625.003494] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 6628.004856] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6628.153505] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6629.515352] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 6631.472642] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6632.838328] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6635.319333] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6637.584058] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 6639.519022] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6659.042812] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6659.044260] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6659.045634] LustreError: Skipped 1 previous similar message [ 6659.050642] Lustre: Skipped 1 previous similar message [ 6659.052860] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6659.055921] Lustre: Skipped 1 previous similar message [ 6660.145092] LustreError: 273534:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6660.148258] LustreError: 273534:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6660.186874] Lustre: server umount lustre-OST0000 complete [ 6664.161803] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6664.161872] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6664.167603] Lustre: Skipped 1 previous similar message [ 6667.583496] LustreError: 273735:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6667.586034] LustreError: 273735:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6667.610899] Lustre: server umount lustre-MDT0000 complete [ 6669.019366] LustreError: 271295:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769761567 with bad export cookie 3513108834179618869 [ 6669.021531] LustreError: 271989:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000bd2:0x1:0x0]: rc = -5 [ 6669.022655] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6669.023965] LustreError: 271295:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6669.139289] Lustre: server umount lustre-MDT0001 complete [ 6672.671885] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing unload_modules_local [ 6673.669424] Key type lgssc unregistered [ 6673.799375] LNet: 274468:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6673.802654] LNetError: 274468:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6673.813387] LNet: Removed LNI 192.168.201.107@tcp [ 6674.095205] Key type .llcrypt unregistered [ 6674.096163] Key type ._llcrypt unregistered [ 6682.525134] Key type ._llcrypt registered [ 6682.526909] Key type .llcrypt registered [ 6682.566085] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_hostid [ 6688.280552] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 6688.573450] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6688.611093] alg: No test for adler32 (adler32-zlib) [ 6689.452737] Lustre: Lustre: Build Version: 2.17.50_86_g2f7fa4d [ 6689.533428] LNet: Added LNI 192.168.201.107@tcp [8/256/0/180] [ 6691.112162] Key type lgssc registered [ 6691.440490] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6705.760193] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6705.765389] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6706.846633] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6706.855983] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6706.891051] Lustre: lustre-MDT0000: new disk, initializing [ 6706.914364] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6706.921163] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6708.148782] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 6712.339695] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6712.381031] Lustre: 278227:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6712.406377] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6712.408888] Lustre: Skipped 1 previous similar message [ 6712.443944] Lustre: lustre-MDT0001: new disk, initializing [ 6712.484599] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6712.501660] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6712.505127] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6713.766965] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 6716.770948] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6717.914480] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6720.120566] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6720.215826] Lustre: lustre-OST2710: new disk, initializing [ 6720.218526] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 6720.246576] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 6721.532221] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 6721.535754] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 6721.580792] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 6722.095714] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 6725.076584] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 6727.271186] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6727.332099] Lustre: lustre-OST03e8: new disk, initializing [ 6727.334387] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 6727.366804] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 6728.645228] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 6728.648863] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 6728.699110] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 6729.228465] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 6730.832151] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 6733.302507] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 6733.378679] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6734.581916] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 6734.659546] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6735.881900] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 6735.957225] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6737.249957] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 6737.318624] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6738.400737] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6738.402949] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6738.407169] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6738.915118] Lustre: lustre-MDT0000-lwp-OST03e8: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6738.915844] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6738.919083] Lustre: Skipped 1 previous similar message [ 6744.033935] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6744.035838] Lustre: Skipped 4 previous similar messages [ 6744.383844] LustreError: 282556:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6744.416571] Lustre: server umount lustre-MDT0000 complete [ 6745.598102] LustreError: 278217:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769761643 with bad export cookie 13472400330660317892 [ 6745.600954] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6745.601645] LustreError: 278217:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6745.660501] LustreError: 282758:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6745.662355] LustreError: 282758:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6745.709337] Lustre: server umount lustre-MDT0001 complete [ 6757.364116] LustreError: 282958:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6757.366281] LustreError: 282958:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6757.385371] Lustre: server umount lustre-OST2710 complete [ 6768.820131] LustreError: 283160:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 6768.822615] LustreError: 283160:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6768.871184] Lustre: server umount lustre-OST03e8 complete [ 6774.570120] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_hostid [ 6777.452392] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 6788.500345] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6788.583203] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6788.593462] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6788.628197] Lustre: lustre-MDT0000: new disk, initializing [ 6788.650233] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6788.657416] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6789.920504] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 6793.910842] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6793.949796] Lustre: 285939:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6794.048529] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6795.219376] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 6798.134859] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6799.251968] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6801.377176] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6801.475114] Lustre: lustre-OST0000: new disk, initializing [ 6801.476578] Lustre: Skipped 1 previous similar message [ 6801.478683] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6801.480952] Lustre: Skipped 2 previous similar messages [ 6801.503886] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6801.506327] Lustre: Skipped 1 previous similar message [ 6803.281023] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 6803.642713] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6803.645576] Lustre: Skipped 1 previous similar message [ 6803.647719] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6803.675327] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6806.230721] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6807.482082] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6807.544961] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6808.772459] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6808.837272] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6814.177569] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6814.179271] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6814.180608] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6814.183339] Lustre: Skipped 2 previous similar messages [ 6814.186779] Lustre: Skipped 2 previous similar messages [ 6815.532120] LustreError: 288869:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6815.534298] LustreError: 288869:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6815.570686] Lustre: server umount lustre-OST0000 complete [ 6819.297499] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6819.298238] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6819.300982] Lustre: Skipped 1 previous similar message [ 6819.303030] Lustre: Skipped 1 previous similar message [ 6829.536985] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6829.538909] Lustre: Skipped 3 previous similar messages [ 6831.072088] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6831.102102] LustreError: 289071:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6831.104079] LustreError: 289071:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6831.128885] Lustre: server umount lustre-MDT0000 complete [ 6832.281962] LustreError: 287679:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769761730 with bad export cookie 13472400330660319558 [ 6832.284452] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6832.285078] LustreError: 287679:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6832.385775] Lustre: server umount lustre-MDT0001 complete [ 6835.244143] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 03:28:53 (1769761733) [ 6835.733143] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 6836.259301] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 03:28:54 (1769761734) [ 6837.250754] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 6845.706761] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6845.789356] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6845.797110] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6845.854187] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6847.108408] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 6849.933937] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6849.962964] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6849.971840] Lustre: 291367:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6851.366011] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 6853.153601] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6854.308680] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6867.496471] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6867.565137] Lustre: Denying initial registration attempt from nid 192.168.201.107@tcp, specified as failover [ 6867.568347] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 6867.571372] LustreError: 292979:0:(tgt_mount.c:2579:server_fill_super()) Unable to start targets: -99 [ 6867.573709] LustreError: 292979:0:(tgt_mount.c:2066:server_put_super()) no obd lustre-OST0000 [ 6867.575988] LustreError: 292979:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 6867.597350] Lustre: server umount lustre-OST0000 complete [ 6867.598724] LustreError: 292979:0:(super25.c:190:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 6870.496745] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6870.499021] LustreError: Skipped 1 previous similar message [ 6870.500261] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6870.504122] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6875.838065] LustreError: 293347:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 6875.840232] LustreError: 293347:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 6875.864119] Lustre: server umount lustre-MDT0000 complete [ 6877.024094] LustreError: 291359:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769761775 with bad export cookie 13472400330660320426 [ 6877.026856] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6877.026890] LustreError: 291359:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6877.134304] Lustre: server umount lustre-MDT0001 complete [ 6880.498156] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing unload_modules_local [ 6881.438346] Key type lgssc unregistered [ 6881.565401] LNet: 294080:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6881.568238] LNetError: 294080:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6881.576321] LNet: Removed LNI 192.168.201.107@tcp [ 6881.862296] Key type .llcrypt unregistered [ 6881.863685] Key type ._llcrypt unregistered [ 6885.263266] Key type ._llcrypt registered [ 6885.264318] Key type .llcrypt registered [ 6885.303217] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 6885.759438] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6885.766500] alg: No test for adler32 (adler32-zlib) [ 6886.609421] Lustre: Lustre: Build Version: 2.17.50_86_g2f7fa4d [ 6886.689996] LNet: Added LNI 192.168.201.107@tcp [8/256/0/180] [ 6888.272205] Key type lgssc registered [ 6888.640078] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6923.098683] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 6925.904712] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6925.910687] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6927.004379] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6927.013436] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6927.076400] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6928.292782] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 6930.939146] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6930.969341] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6930.978638] Lustre: 296788:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6931.062089] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6932.268676] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 6933.939673] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6935.049661] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6948.169601] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6948.232299] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6948.289616] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6950.012448] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 6951.717687] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6952.410203] LustreError: 299054:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6952.447603] Lustre: server umount lustre-OST0000 complete [ 6957.024892] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6957.027743] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6957.031525] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6958.062793] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6958.062812] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6958.068187] Lustre: Skipped 1 previous similar message [ 6959.741899] LustreError: 299254:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 6959.743709] LustreError: 299254:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6959.768150] Lustre: server umount lustre-MDT0000 complete [ 6960.972227] LustreError: 297514:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769761859 with bad export cookie 17971357692553156350 [ 6960.975722] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6960.975806] LustreError: 297514:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6961.039540] LustreError: 299460:0:(obd_class.h:479:obd_check_dev()) Device 7 not setup [ 6961.041989] LustreError: 299460:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6961.096131] Lustre: server umount lustre-MDT0001 complete [ 6964.592688] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing unload_modules_local [ 6965.606651] Key type lgssc unregistered [ 6965.731498] LNet: 299991:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6965.734111] LNetError: 299991:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6965.740307] LNet: Removed LNI 192.168.201.107@tcp [ 6966.028443] Key type .llcrypt unregistered [ 6966.029602] Key type ._llcrypt unregistered [ 6974.691689] Key type ._llcrypt registered [ 6974.693022] Key type .llcrypt registered [ 6974.729564] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 6975.081556] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6975.109062] alg: No test for adler32 (adler32-zlib) [ 6975.956123] Lustre: Lustre: Build Version: 2.17.50_86_g2f7fa4d [ 6976.032375] LNet: Added LNI 192.168.201.107@tcp [8/256/0/180] [ 6977.608109] Key type lgssc registered [ 6977.995072] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6980.436361] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6980.441467] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6981.585819] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6982.819210] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 6985.477332] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6985.595238] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6986.785532] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 6988.405258] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6989.512822] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6991.571547] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6991.678214] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6993.370708] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 6994.943935] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7001.058548] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7001.058661] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7001.060093] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7001.061680] Lustre: Skipped 1 previous similar message [ 7003.842932] LustreError: 303607:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7003.870308] Lustre: server umount lustre-MDT0000 complete [ 7005.036417] LustreError: 301347:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769761903 with bad export cookie 6280453135983351174 [ 7005.039283] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7005.039649] LustreError: 301347:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7006.176869] LustreError: 301360:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7006.177175] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7006.184233] LustreError: 301360:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7006.187074] Lustre: Skipped 1 previous similar message [ 7006.187640] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7006.194153] Lustre: Skipped 3 previous similar messages [ 7010.208706] LustreError: 302976:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7010.208896] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7010.213237] LustreError: 302976:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7010.214818] Lustre: Skipped 1 previous similar message [ 7011.129510] LustreError: 303808:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 7011.131213] LustreError: 303808:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7011.170890] Lustre: server umount lustre-MDT0001 complete [ 7013.128302] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 7015.723346] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7030.752686] LustreError: 300674:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000041094ef7 x1855729826812800/t0(0) o250->MGC192.168.201.107@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7030.860153] LustreError: 302967:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7030.883480] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7030.910618] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:161) [ 7032.081183] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 7034.704147] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7035.877926] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7036.018899] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 7037.660911] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7038.159296] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7039.218756] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7039.973678] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7047.863826] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 8 sec [ 7050.210108] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7050.210190] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7050.212681] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7050.215393] Lustre: Skipped 1 previous similar message [ 7050.218640] Lustre: Skipped 1 previous similar message [ 7055.087062] LustreError: 306113:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7055.089764] LustreError: 306113:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7055.124083] Lustre: server umount lustre-OST0000 complete [ 7060.449472] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7060.449647] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7060.450322] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7060.458460] Lustre: Skipped 1 previous similar message [ 7062.394190] LustreError: 306314:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7062.396561] LustreError: 306314:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7062.419372] Lustre: server umount lustre-MDT0000 complete [ 7063.608166] LustreError: 303809:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769761961 with bad export cookie 6280453135983358328 [ 7063.610611] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7063.612712] LustreError: 303809:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7063.722135] Lustre: server umount lustre-MDT0001 complete [ 7066.930068] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing unload_modules_local [ 7067.880420] Key type lgssc unregistered [ 7068.000272] LNet: 307046:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7068.002292] LNetError: 307046:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7068.009272] LNet: Removed LNI 192.168.201.107@tcp [ 7068.319635] Key type .llcrypt unregistered [ 7068.320873] Key type ._llcrypt unregistered [ 7076.971369] Key type ._llcrypt registered [ 7076.972405] Key type .llcrypt registered [ 7077.007877] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 7077.317175] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7077.352030] alg: No test for adler32 (adler32-zlib) [ 7078.197726] Lustre: Lustre: Build Version: 2.17.50_86_g2f7fa4d [ 7078.285165] LNet: Added LNI 192.168.201.107@tcp [8/256/0/180] [ 7079.872203] Key type lgssc registered [ 7080.265651] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7082.779140] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7082.784701] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7083.936642] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7085.219928] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 7087.907131] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7088.033203] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7089.377317] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 7091.129624] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7092.256865] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7098.338041] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7098.338206] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7098.338669] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7098.342510] Lustre: Skipped 1 previous similar message [ 7099.361257] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7099.363562] Lustre: Skipped 2 previous similar messages [ 7099.392329] LustreError: 309807:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7099.415040] Lustre: server umount lustre-MDT0000 complete [ 7100.598141] LustreError: 308401:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769761998 with bad export cookie 18412261469045524432 [ 7100.601227] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7100.602034] LustreError: 308401:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7100.666227] LustreError: 310010:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 7100.668008] LustreError: 310010:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7100.721295] Lustre: server umount lustre-MDT0001 complete [ 7105.242155] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7105.335527] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7105.345835] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7105.419796] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7106.598991] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 7109.259174] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7109.293320] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7109.303556] Lustre: 310778:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7109.393229] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7110.634028] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 7112.376904] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7113.498580] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7115.954945] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7116.024785] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 7116.027778] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 7116.030963] LustreError: 312435:0:(tgt_mount.c:2579:server_fill_super()) Unable to start targets: -2 [ 7116.033894] LustreError: 312435:0:(tgt_mount.c:2066:server_put_super()) no obd lustre-OST0000 [ 7116.036263] LustreError: 312435:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 7116.061112] Lustre: server umount lustre-OST0000 complete [ 7116.062417] LustreError: 312435:0:(super25.c:190:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 7118.882040] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7118.957729] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7119.024946] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7120.725916] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:193) [ 7120.968953] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 7122.833558] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7125.358073] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7125.387470] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7125.420808] Lustre: lustre-OST0001: new disk, initializing [ 7125.423211] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7127.403837] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 7129.236979] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7129.287543] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7129.291604] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7129.330413] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7134.692991] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7134.693521] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7134.694300] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7134.696679] Lustre: Skipped 1 previous similar message [ 7134.701301] LustreError: Skipped 1 previous similar message [ 7136.238111] LustreError: 314550:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 7136.240375] LustreError: 314550:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 7136.260556] Lustre: server umount lustre-OST0001 complete [ 7139.810492] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7139.812058] LustreError: 312902:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7139.812690] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7139.812696] Lustre: Skipped 1 previous similar message [ 7139.818295] Lustre: Skipped 1 previous similar message [ 7139.823059] LustreError: 312902:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7143.727181] LustreError: 314752:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7143.729939] LustreError: 314752:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7143.766720] Lustre: server umount lustre-OST0000 complete [ 7145.440789] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7145.442687] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7145.446433] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7145.448669] Lustre: Skipped 1 previous similar message [ 7150.049786] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7151.167392] LustreError: 314953:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 7151.169625] LustreError: 314953:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7151.198175] Lustre: server umount lustre-MDT0000 complete [ 7152.425281] LustreError: 313861:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769762050 with bad export cookie 18412261469045524971 [ 7152.426498] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7152.428960] LustreError: 313861:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7152.538676] Lustre: server umount lustre-MDT0001 complete [ 7155.810113] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing unload_modules_local [ 7156.761726] Key type lgssc unregistered [ 7156.873349] LNet: 315686:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7156.876519] LNetError: 315686:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7156.886298] LNet: Removed LNI 192.168.201.107@tcp [ 7157.167457] Key type .llcrypt unregistered [ 7157.168529] Key type ._llcrypt unregistered [ 7185.808849] Key type ._llcrypt registered [ 7185.809858] Key type .llcrypt registered [ 7185.843835] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_hostid [ 7191.504114] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 7191.809409] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7191.832058] alg: No test for adler32 (adler32-zlib) [ 7192.677894] Lustre: Lustre: Build Version: 2.17.50_86_g2f7fa4d [ 7192.752891] LNet: Added LNI 192.168.201.107@tcp [8/256/0/180] [ 7194.320116] Key type lgssc registered [ 7194.651942] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7204.635804] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7204.641293] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7205.718120] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7205.731830] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7205.773355] Lustre: lustre-MDT0000: new disk, initializing [ 7205.798180] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7205.806156] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7206.961361] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 7210.743610] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7210.776486] Lustre: 320551:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7210.796096] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7210.798396] Lustre: Skipped 1 previous similar message [ 7210.832471] Lustre: lustre-MDT0001: new disk, initializing [ 7210.861962] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7210.873131] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7210.875571] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7211.956437] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 7214.980996] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7216.145323] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7218.274509] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7218.367793] Lustre: lustre-OST0000: new disk, initializing [ 7218.369339] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7218.396319] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7220.092879] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 7221.620361] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7221.622911] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7221.646714] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7222.960666] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7224.142987] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7224.200760] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7225.392390] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7225.454749] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7226.849736] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7226.850910] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7226.851536] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7226.852308] LustreError: Skipped 1 previous similar message [ 7226.858695] Lustre: Skipped 1 previous similar message [ 7228.080103] LustreError: 323481:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7228.123371] Lustre: server umount lustre-OST0000 complete [ 7231.456775] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7231.458554] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7231.461739] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7231.463600] Lustre: Skipped 1 previous similar message [ 7235.515727] LustreError: 323682:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 7235.517344] LustreError: 323682:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7235.544835] Lustre: server umount lustre-MDT0000 complete [ 7236.728201] LustreError: 321328:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769762134 with bad export cookie 10233879258028643676 [ 7236.728815] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7236.731856] LustreError: 321328:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7236.793909] LustreError: 323884:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7236.795421] LustreError: 323884:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7236.841521] Lustre: server umount lustre-MDT0001 complete [ 7239.383760] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 03:35:37 (1769762137) [ 7241.590455] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 03:35:39 (1769762139) [ 7244.226556] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7244.388178] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7245.564757] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 7248.122513] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7249.337131] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 7250.965578] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7252.020506] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7253.981561] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7254.074332] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7254.076859] Lustre: Skipped 1 previous similar message [ 7255.730981] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 7257.369254] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7261.517076] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 7261.982104] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 7262.427692] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 7262.919798] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 7263.512721] Lustre: Failing over lustre-MDT0000 [ 7263.541697] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.7@tcp (stopping) [ 7263.544392] Lustre: Skipped 1 previous similar message [ 7263.680978] LustreError: 327328:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7263.682629] LustreError: 327328:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 7263.703272] Lustre: server umount lustre-MDT0000 complete [ 7263.713083] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7263.715378] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7263.718617] Lustre: Skipped 1 previous similar message [ 7263.720352] LustreError: 324895:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7263.725432] LustreError: 324895:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7268.321538] LustreError: 324895:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7268.323387] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7273.441854] LustreError: 327445:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7273.447082] LustreError: 327445:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 7276.542331] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7276.575404] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7276.656822] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7276.679433] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 7277.879460] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 7278.894474] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 7282.152435] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7282.161406] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 7282.178476] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 7283.206490] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7283.711195] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7285.525128] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 7286.162987] Lustre: Failing over lustre-MDT0000 [ 7286.284880] LustreError: 328410:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7286.287678] LustreError: 328410:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7286.315190] Lustre: server umount lustre-MDT0000 complete [ 7287.266128] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7287.267321] LustreError: 325605:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7287.271055] Lustre: Skipped 3 previous similar messages [ 7287.275237] LustreError: 325605:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7287.277681] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7292.385205] LustreError: 324896:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7292.390074] LustreError: 324896:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7298.990123] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7299.027243] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7299.109598] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7299.137506] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 7299.374980] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 7300.235062] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 7304.169765] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7304.171405] Lustre: Skipped 2 previous similar messages [ 7304.177268] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 7304.193235] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 7305.295249] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7305.844230] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7309.282607] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7309.283846] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7309.284389] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7309.289640] Lustre: Skipped 1 previous similar message [ 7314.174502] LustreError: 329442:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7314.176559] LustreError: 329442:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7314.205959] Lustre: server umount lustre-MDT0000 complete [ 7314.402754] LustreError: 324895:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7314.407974] LustreError: 324895:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 7315.374267] LustreError: 324875:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769762213 with bad export cookie 10233879258028646329 [ 7315.377488] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7315.377838] LustreError: 324875:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7315.472542] Lustre: server umount lustre-MDT0001 complete [ 7325.919277] Lustre: server umount lustre-OST0000 complete [ 7330.825251] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7330.955027] LustreError: 330671:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7332.157198] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 7334.886657] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7335.002113] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7335.004555] Lustre: Skipped 1 previous similar message [ 7336.280190] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 7337.969898] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7339.047190] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7341.213573] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7343.156223] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 7344.852776] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7356.709982] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 7362.018753] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7362.020051] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7362.024443] Lustre: Skipped 2 previous similar messages [ 7362.025763] Lustre: Skipped 2 previous similar messages [ 7362.029953] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7363.506019] LustreError: 332934:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7363.507893] LustreError: 332934:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 7363.543246] Lustre: server umount lustre-OST0000 complete [ 7366.814351] Lustre: server umount lustre-MDT0000 complete [ 7367.137735] LustreError: 330670:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7367.142789] LustreError: 330670:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7368.089277] LustreError: 330652:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769762266 with bad export cookie 10233879258028647078 [ 7368.093061] LustreError: 330652:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7368.094404] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7368.209819] Lustre: server umount lustre-MDT0001 complete [ 7371.796202] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing unload_modules_local [ 7372.771662] Key type lgssc unregistered [ 7372.889333] LNet: 333868:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7372.891783] LNetError: 333868:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7372.899396] LNet: Removed LNI 192.168.201.107@tcp [ 7373.197395] Key type .llcrypt unregistered [ 7373.198485] Key type ._llcrypt unregistered [ 7382.151781] Key type ._llcrypt registered [ 7382.152647] Key type .llcrypt registered [ 7382.185295] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 7382.598862] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7382.609182] alg: No test for adler32 (adler32-zlib) [ 7383.452523] Lustre: Lustre: Build Version: 2.17.50_86_g2f7fa4d [ 7383.528670] LNet: Added LNI 192.168.201.107@tcp [8/256/0/180] [ 7385.104154] Key type lgssc registered [ 7385.444986] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7387.826598] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7387.834254] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7388.972434] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7390.160963] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 7392.906509] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7393.031976] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7394.257236] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 7395.901819] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7397.007622] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7399.041776] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7399.140496] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7400.167020] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 7400.835447] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 7402.435586] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7408.610830] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7408.612081] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7408.614308] Lustre: Skipped 2 previous similar messages [ 7408.614629] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7408.615933] Lustre: Skipped 2 previous similar messages [ 7413.182472] LustreError: 337581:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7413.206783] Lustre: server umount lustre-MDT0000 complete [ 7413.730138] LustreError: 335366:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7413.736896] LustreError: 335366:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 7414.341232] LustreError: 335323:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769762312 with bad export cookie 11175855649466903923 [ 7414.344829] LustreError: 335323:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7414.344838] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7414.403049] LustreError: 337784:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 7414.405089] LustreError: 337784:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7414.448047] Lustre: server umount lustre-MDT0001 complete [ 7425.843537] LustreError: 337985:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 7425.845575] LustreError: 337985:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7425.884445] Lustre: server umount lustre-OST0000 complete [ 7430.032801] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7430.182382] LustreError: 338714:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7430.205037] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7431.438176] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 7434.120636] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7434.248712] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7435.487856] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 7437.159778] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7438.345579] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7440.437452] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7442.317699] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 7444.082891] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7454.886324] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 7470.048674] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7470.050290] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7470.053119] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7472.507547] LustreError: 341079:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7472.509823] LustreError: 341079:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7472.537091] Lustre: server umount lustre-MDT0000 complete [ 7473.741137] LustreError: 338740:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769762371 with bad export cookie 11175855649466905022 [ 7473.744500] LustreError: 338740:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7473.745080] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7473.848210] Lustre: server umount lustre-MDT0001 complete [ 7485.232826] LustreError: 341486:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 7485.235349] LustreError: 341486:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 7485.276799] Lustre: server umount lustre-OST0000 complete [ 7489.923393] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7490.046159] LustreError: 342312:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7490.063882] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7490.065741] Lustre: Skipped 1 previous similar message [ 7491.215948] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 7493.894739] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7495.167796] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 7496.839975] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7497.981640] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7500.054054] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7501.224903] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 7501.954468] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 7503.611102] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7506.401049] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7506.405111] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7506.411589] Lustre: Skipped 2 previous similar messages [ 7506.415153] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7506.418260] Lustre: Skipped 2 previous similar messages [ 7509.475721] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7509.475780] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7509.476156] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7509.477483] Lustre: Skipped 1 previous similar message [ 7510.767100] LustreError: 344548:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7510.768991] LustreError: 344548:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7510.808523] Lustre: server umount lustre-OST0000 complete [ 7514.594019] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7514.594224] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7514.600482] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7514.602418] Lustre: Skipped 1 previous similar message [ 7524.833613] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7524.835288] Lustre: Skipped 3 previous similar messages [ 7526.368137] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7526.426426] Lustre: server umount lustre-MDT0000 complete [ 7527.637687] LustreError: 342293:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769762425 with bad export cookie 11175855649466906030 [ 7527.642037] LustreError: 342293:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7527.642460] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7527.715815] LustreError: 344956:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 7527.718883] LustreError: 344956:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 7527.777638] Lustre: server umount lustre-MDT0001 complete [ 7531.663117] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing unload_modules_local [ 7532.785969] Key type lgssc unregistered [ 7532.920405] LNet: 345487:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7532.922717] LNetError: 345487:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7532.936392] LNet: Removed LNI 192.168.201.107@tcp [ 7533.232249] Key type .llcrypt unregistered [ 7533.233880] Key type ._llcrypt unregistered [ 7542.216873] Key type ._llcrypt registered [ 7542.218291] Key type .llcrypt registered [ 7542.253853] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 7542.659135] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7542.667804] alg: No test for adler32 (adler32-zlib) [ 7543.512163] Lustre: Lustre: Build Version: 2.17.50_86_g2f7fa4d [ 7543.587896] LNet: Added LNI 192.168.201.107@tcp [8/256/0/180] [ 7545.160270] Key type lgssc registered [ 7545.522977] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7548.103216] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7548.107992] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7548.110805] LustreError: 346883:0:(osd_handler.c:8564:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 7548.118144] LustreError: 346883:0:(obd_config.c:859:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 7548.121358] LustreError: 346883:0:(obd_mount.c:193:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 7548.124518] LustreError: 346883:0:(tgt_mount.c:2547:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 7548.128447] LustreError: 346883:0:(super25.c:190:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7550.449714] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7550.451928] LustreError: 347279:0:(osd_handler.c:8564:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 7550.459903] LustreError: 347279:0:(obd_config.c:859:class_setup()) setup lustre-OST0000-osd failed (-22) [ 7550.462221] LustreError: 347279:0:(obd_mount.c:193:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 7550.464113] LustreError: 347279:0:(tgt_mount.c:2547:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 7550.466328] LustreError: 347279:0:(super25.c:190:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7556.315416] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing unload_modules_local [ 7557.259343] Key type lgssc unregistered [ 7557.385307] LNet: 348262:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7557.387057] LNetError: 348262:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7557.395276] LNet: Removed LNI 192.168.201.107@tcp [ 7557.650536] Key type .llcrypt unregistered [ 7557.651804] Key type ._llcrypt unregistered [ 7562.950809] Key type ._llcrypt registered [ 7562.951611] Key type .llcrypt registered [ 7562.982100] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_hostid [ 7568.635560] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 7568.924572] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7568.933164] alg: No test for adler32 (adler32-zlib) [ 7569.787192] Lustre: Lustre: Build Version: 2.17.50_86_g2f7fa4d [ 7569.859079] LNet: Added LNI 192.168.201.107@tcp [8/256/0/180] [ 7571.432127] Key type lgssc registered [ 7571.781690] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7582.522247] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7582.527046] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7583.623097] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7583.637323] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7583.676209] Lustre: lustre-MDT0000: new disk, initializing [ 7583.700633] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7583.707575] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7584.986895] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 7589.116629] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7589.156687] Lustre: 351354:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7589.177531] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7589.179077] Lustre: Skipped 1 previous similar message [ 7589.212483] Lustre: lustre-MDT0001: new disk, initializing [ 7589.243348] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7589.254849] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7589.258902] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7590.560609] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 7593.673485] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7594.938870] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7597.317097] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7597.451461] Lustre: lustre-OST0000: new disk, initializing [ 7597.454176] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7597.492235] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7598.968356] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7598.973853] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7599.019600] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7599.550882] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 7602.612508] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7603.868061] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7603.936883] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7605.156651] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7605.228994] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7609.314206] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7609.316167] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7609.317274] LustreError: Skipped 1 previous similar message [ 7609.322588] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7609.325305] Lustre: Skipped 1 previous similar message [ 7614.436473] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7614.439192] Lustre: Skipped 1 previous similar message [ 7619.554810] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7619.556543] Lustre: Skipped 1 previous similar message [ 7620.064145] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7620.086163] LustreError: 354282:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7620.132945] Lustre: server umount lustre-OST0000 complete [ 7624.673913] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7624.674539] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7624.677215] Lustre: Skipped 2 previous similar messages [ 7624.680795] Lustre: Skipped 1 previous similar message [ 7627.523106] LustreError: 354486:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 7627.525674] LustreError: 354486:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7627.554606] Lustre: server umount lustre-MDT0000 complete [ 7628.827072] LustreError: 354487:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769762526 with bad export cookie 13529944002186278963 [ 7628.830487] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7628.831477] LustreError: 354487:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7628.896584] LustreError: 354688:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7628.898488] LustreError: 354688:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7628.961969] Lustre: server umount lustre-MDT0001 complete [ 7631.805580] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 7632.340664] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 03:42:10 (1769762530) [ 7634.501719] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7634.679644] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7635.959510] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 7638.668969] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7638.791674] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7640.135221] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 7641.885466] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7643.084658] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7645.267221] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7647.155666] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 7648.920378] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7651.139776] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7651.210289] Lustre: lustre-OST0001: new disk, initializing [ 7651.211987] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7651.247717] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7651.251232] Lustre: Skipped 1 previous similar message [ 7653.122578] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 7656.208727] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7658.498320] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7658.502565] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7658.527325] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7663.585103] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7663.585343] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7663.587269] LustreError: Skipped 1 previous similar message [ 7663.587735] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7663.590951] Lustre: Skipped 1 previous similar message [ 7663.594990] Lustre: Skipped 1 previous similar message [ 7668.077835] LustreError: 358596:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 7668.081399] LustreError: 358596:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 7668.101691] Lustre: server umount lustre-OST0001 complete [ 7669.731742] LustreError: 357912:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7669.736271] LustreError: 357912:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7674.855935] LustreError: 357912:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7674.860110] LustreError: 357912:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7679.977060] LustreError: 357912:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7679.982071] LustreError: 357912:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7684.577183] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7684.580929] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7684.588430] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7684.591827] Lustre: Skipped 3 previous similar messages [ 7685.097903] LustreError: 356996:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7685.112446] LustreError: 356996:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7686.258104] LustreError: 358819:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7686.259908] LustreError: 358819:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7686.292478] Lustre: server umount lustre-OST0000 complete [ 7690.210827] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7690.211220] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7690.216825] Lustre: Skipped 2 previous similar messages [ 7690.219803] LustreError: Skipped 1 previous similar message [ 7693.856280] Lustre: server umount lustre-MDT0000 complete [ 7695.106950] LustreError: 355385:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769762593 with bad export cookie 13529944002186279831 [ 7695.111320] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7695.112404] LustreError: 355385:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7701.256168] LustreError: 359222:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 7701.261037] LustreError: 359222:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 7701.328669] Lustre: server umount lustre-MDT0001 complete [ 7704.836709] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing unload_modules_local [ 7705.896424] Key type lgssc unregistered [ 7706.034509] LNet: 359754:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7706.036907] LNetError: 359754:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7706.043470] LNet: Removed LNI 192.168.201.107@tcp [ 7706.340065] Key type .llcrypt unregistered [ 7706.341094] Key type ._llcrypt unregistered [ 7724.092856] Key type ._llcrypt registered [ 7724.093888] Key type .llcrypt registered [ 7724.134979] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 7724.499457] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7724.537352] alg: No test for adler32 (adler32-zlib) [ 7725.384555] Lustre: Lustre: Build Version: 2.17.50_86_g2f7fa4d [ 7725.472200] LNet: Added LNI 192.168.201.107@tcp [8/256/0/180] [ 7727.048160] Key type lgssc registered [ 7727.388964] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7729.920937] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7729.927956] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7731.023972] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7731.032892] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7731.092850] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7732.446744] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 7735.250903] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7735.282367] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7735.292431] Lustre: 362507:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7735.393529] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7736.715989] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 7738.444966] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7739.599498] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7740.385096] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7740.387369] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7740.391337] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7743.969294] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7743.969881] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7746.367095] LustreError: 363869:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 7746.394236] Lustre: server umount lustre-MDT0000 complete [ 7747.618429] LustreError: 363871:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769762645 with bad export cookie 146273459116317148 [ 7747.622277] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7747.623328] LustreError: 363871:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7747.682077] LustreError: 364072:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 7747.684863] LustreError: 364072:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7747.734566] Lustre: server umount lustre-MDT0001 complete [ 7750.842821] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 03:44:09 (1769762649) [ 7753.032965] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7753.205271] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7754.485808] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 7757.261322] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7757.375537] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7758.658313] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 7760.449664] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7761.674515] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7763.925240] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7763.991546] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7764.051667] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7765.971612] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 7767.694096] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7776.479452] Lustre: Permanently deactivating lustre-OST0000 [ 7776.484469] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7778.546369] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7779.664888] LustreError: 367335:0:(mgs_llog.c:1942:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [ 7779.668326] LustreError: 367335:0:(mgs_llog.c:1997:mgs_replace_nids()) Only MGS is allowed to be started [ 7779.670603] LustreError: 367335:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 7780.269506] LustreError: 367384:0:(mgs_llog.c:1997:mgs_replace_nids()) Only MGS is allowed to be started [ 7780.271752] LustreError: 367384:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 7786.991105] LustreError: 367481:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7786.992956] LustreError: 367481:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7787.030642] Lustre: server umount lustre-OST0000 complete [ 7787.936214] LustreError: 367633:0:(mgs_llog.c:1997:mgs_replace_nids()) Only MGS is allowed to be started [ 7787.940109] LustreError: 367633:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 7791.073692] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7791.074319] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7791.077104] Lustre: Skipped 1 previous similar message [ 7791.080686] Lustre: Skipped 2 previous similar messages [ 7796.193106] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7796.195067] Lustre: Skipped 1 previous similar message [ 7798.243446] LustreError: 367731:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 7798.245408] LustreError: 367731:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7798.265742] Lustre: server umount lustre-MDT0000 complete [ 7799.455278] LustreError: 364813:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769762697 with bad export cookie 146273459116317722 [ 7799.457503] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7799.459625] LustreError: 364813:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7808.995834] LustreError: 367934:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7808.998044] LustreError: 367934:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7809.053321] Lustre: server umount lustre-MDT0001 complete [ 7811.866236] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7813.106056] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 7813.978543] LustreError: 368775:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [ 7815.965631] Lustre: 369113:0:(mgs_llog.c:1709:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.201.107@tcp [ 7816.667855] Lustre: server umount MGS complete [ 7820.282680] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7820.420599] LustreError: 369838:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7820.425078] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7820.425892] LustreError: 369838:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7820.427741] Lustre: Skipped 1 previous similar message [ 7820.445370] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7821.691416] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 7824.556798] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7824.653139] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7826.015502] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 7827.827755] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7829.095033] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7829.743897] Lustre: Permanently reactivating lustre-OST0000 [ 7829.749300] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7829.751493] Lustre: Skipped 2 previous similar messages [ 7850.464434] LustreError: 371240:0:(osp_dev.c:692:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [ 7852.788444] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7852.891924] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7852.893967] Lustre: Skipped 1 previous similar message [ 7854.654899] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 7856.346463] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7870.436070] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7870.437483] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7870.437669] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7870.439353] Lustre: Skipped 1 previous similar message [ 7870.444568] LustreError: Skipped 1 previous similar message [ 7874.028271] LustreError: 373024:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7874.031625] LustreError: 373024:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 7874.071210] Lustre: server umount lustre-OST0000 complete [ 7875.553440] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7880.672986] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7880.675301] Lustre: Skipped 4 previous similar messages [ 7881.432515] Lustre: server umount lustre-MDT0000 complete [ 7882.675391] LustreError: 369818:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769762780 with bad export cookie 146273459116318744 [ 7882.678477] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7882.681319] LustreError: 369818:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7882.814760] Lustre: server umount lustre-MDT0001 complete [ 7886.235374] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing unload_modules_local [ 7887.265887] Key type lgssc unregistered [ 7887.394396] LNet: 373961:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7887.397135] LNetError: 373961:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7887.403339] LNet: Removed LNI 192.168.201.107@tcp [ 7887.704807] Key type .llcrypt unregistered [ 7887.706643] Key type ._llcrypt unregistered [ 7893.475556] Key type ._llcrypt registered [ 7893.476724] Key type .llcrypt registered [ 7893.518618] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_hostid [ 7899.821028] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 7900.147157] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7900.217513] alg: No test for adler32 (adler32-zlib) [ 7901.071315] Lustre: Lustre: Build Version: 2.17.50_86_g2f7fa4d [ 7901.153397] LNet: Added LNI 192.168.201.107@tcp [8/256/0/180] [ 7902.728137] Key type lgssc registered [ 7903.082827] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7913.960517] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7913.964908] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7915.050864] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7915.063498] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7915.100681] Lustre: lustre-MDT0000: new disk, initializing [ 7915.123174] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7915.129371] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7916.375510] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 7920.398685] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7920.441587] Lustre: 377054:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7920.470509] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7920.473227] Lustre: Skipped 1 previous similar message [ 7920.516877] Lustre: lustre-MDT0001: new disk, initializing [ 7920.553881] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7920.566717] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7920.569886] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7921.859445] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 7924.946709] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7926.168422] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7928.412400] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7928.533364] Lustre: lustre-OST0000: new disk, initializing [ 7928.536380] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7928.566887] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7928.644565] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7928.647966] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7928.694879] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7930.506646] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 7933.598292] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7934.927707] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7935.000949] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7936.348131] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7936.422929] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7939.044911] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7939.046135] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7939.047224] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7939.048559] LustreError: Skipped 1 previous similar message [ 7939.052162] Lustre: Skipped 1 previous similar message [ 7944.164436] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7944.166366] Lustre: Skipped 2 previous similar messages [ 7949.285182] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7949.288146] Lustre: Skipped 1 previous similar message [ 7951.328168] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7951.348621] LustreError: 379984:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7951.394607] Lustre: server umount lustre-OST0000 complete [ 7954.401865] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7954.402595] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7954.405055] Lustre: Skipped 1 previous similar message [ 7958.716815] LustreError: 380186:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 7958.718830] LustreError: 380186:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7958.750147] Lustre: server umount lustre-MDT0000 complete [ 7959.908095] LustreError: 380187:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769762858 with bad export cookie 9289655894709730920 [ 7959.912599] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7959.979592] LustreError: 380389:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7959.981310] LustreError: 380389:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7960.036739] Lustre: server umount lustre-MDT0001 complete [ 7962.751394] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [ 7963.291419] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 03:47:41 (1769762861) [ 7965.670959] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7965.811936] LustreError: 381105:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7965.816945] LustreError: 381105:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7965.834152] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7967.061854] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 7969.804660] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7969.939716] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7971.261036] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 7973.110755] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7974.354873] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7976.525397] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7978.506725] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 7980.221110] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7980.882061] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [ 7991.671896] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [ 7995.362734] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7995.363075] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7995.364925] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7995.364930] Lustre: Skipped 1 previous similar message [ 7995.375293] Lustre: Skipped 1 previous similar message [ 7999.214119] LustreError: 383556:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7999.217744] LustreError: 383556:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 7999.259798] Lustre: server umount lustre-OST0000 complete [ 8000.992746] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8000.995518] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8005.601158] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8005.601761] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8005.603160] Lustre: Skipped 2 previous similar messages [ 8006.651484] LustreError: 383758:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 8006.653575] LustreError: 383758:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8006.679873] Lustre: server umount lustre-MDT0000 complete [ 8007.902276] LustreError: 381130:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769762906 with bad export cookie 9289655894709731788 [ 8007.905727] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8007.906173] LustreError: 381130:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 8008.029815] Lustre: server umount lustre-MDT0001 complete [ 8011.609419] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing unload_modules_local [ 8012.755497] Key type lgssc unregistered [ 8012.888424] LNet: 384491:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8012.890735] LNetError: 384491:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8012.898323] LNet: Removed LNI 192.168.201.107@tcp [ 8013.218854] Key type .llcrypt unregistered [ 8013.220082] Key type ._llcrypt unregistered [ 8021.162301] Key type ._llcrypt registered [ 8021.163420] Key type .llcrypt registered [ 8021.197244] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing unload_modules_local [ 8021.277274] Key type .llcrypt unregistered [ 8021.278082] Key type ._llcrypt unregistered [ 8027.991218] Key type ._llcrypt registered [ 8027.992044] Key type .llcrypt registered [ 8028.028729] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 8028.383768] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8028.404151] alg: No test for adler32 (adler32-zlib) [ 8029.255835] Lustre: Lustre: Build Version: 2.17.50_86_g2f7fa4d [ 8029.329243] LNet: Added LNI 192.168.201.107@tcp [8/256/0/180] [ 8030.904105] Key type lgssc registered [ 8031.235255] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8033.500458] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8033.506027] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8034.640262] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8035.764725] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 8038.313495] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8038.414758] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8040.020835] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 8041.559479] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8043.489227] LustreError: 386639:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8043.493095] LustreError: 386639:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 8043.630276] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8043.741695] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8044.775712] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 8044.850524] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 8049.121521] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8049.125057] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8049.131642] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8050.146860] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8050.148614] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8050.149062] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8050.155640] Lustre: Skipped 1 previous similar message [ 8053.614989] LustreError: 388687:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 8053.650156] Lustre: server umount lustre-OST0000 complete [ 8055.265785] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8055.266649] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8055.269604] Lustre: Skipped 1 previous similar message [ 8055.275266] Lustre: Skipped 1 previous similar message [ 8060.385247] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8060.386940] Lustre: Skipped 1 previous similar message [ 8060.990396] LustreError: 388888:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 8060.992911] LustreError: 388888:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8061.019952] Lustre: server umount lustre-MDT0000 complete [ 8062.244091] LustreError: 386619:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769762960 with bad export cookie 15920013152216396181 [ 8062.246879] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8062.247671] LustreError: 386619:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8062.310963] LustreError: 389090:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8062.313250] LustreError: 389090:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8062.357772] Lustre: server umount lustre-MDT0001 complete [ 8065.476343] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing unload_modules_local [ 8066.419696] Key type lgssc unregistered [ 8066.528349] LNet: 389621:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8066.530430] LNetError: 389621:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8066.540275] LNet: Removed LNI 192.168.201.107@tcp [ 8066.809088] Key type .llcrypt unregistered [ 8066.810585] Key type ._llcrypt unregistered [ 8075.243145] Key type ._llcrypt registered [ 8075.244965] Key type .llcrypt registered [ 8075.281285] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 8075.708313] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8075.718820] alg: No test for adler32 (adler32-zlib) [ 8076.572094] Lustre: Lustre: Build Version: 2.17.50_86_g2f7fa4d [ 8076.656961] LNet: Added LNI 192.168.201.107@tcp [8/256/0/180] [ 8078.240148] Key type lgssc registered [ 8078.637277] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8081.302913] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8081.309164] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8096.736347] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 8101.856173] LustreError: 390967:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.107@tcp: failed processing log, type 4: rc = -110 [ 8128.480136] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8130.394528] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 8132.118131] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8134.333815] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8134.503126] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8134.531638] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 8135.737759] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 8138.420319] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8138.540830] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8139.739134] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 8141.386522] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8142.546073] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8144.864874] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8144.867260] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8144.871143] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8148.962682] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8148.963086] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8148.964815] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8150.829860] LustreError: 393247:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 8150.866247] Lustre: server umount lustre-OST0000 complete [ 8154.081918] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8154.082288] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8154.088690] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8154.090804] Lustre: Skipped 2 previous similar messages [ 8158.146408] LustreError: 393449:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 8158.148165] LustreError: 393449:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8158.176618] Lustre: server umount lustre-MDT0000 complete [ 8159.202037] LustreError: 391896:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8159.205770] LustreError: 391896:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 8159.348987] LustreError: 390974:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769763057 with bad export cookie 1669268620557151246 [ 8159.351992] LustreError: 390974:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8159.352646] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8159.418211] LustreError: 393652:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8159.420547] LustreError: 393652:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8159.469082] Lustre: server umount lustre-MDT0001 complete [ 8162.811589] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing unload_modules_local [ 8163.829677] Key type lgssc unregistered [ 8163.956341] LNet: 394183:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8163.959235] LNetError: 394183:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8163.967319] LNet: Removed LNI 192.168.201.107@tcp [ 8164.257684] Key type .llcrypt unregistered [ 8164.258814] Key type ._llcrypt unregistered [ 8172.965635] Key type ._llcrypt registered [ 8172.966580] Key type .llcrypt registered [ 8173.003193] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 8173.443577] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8173.449898] alg: No test for adler32 (adler32-zlib) [ 8174.304649] Lustre: Lustre: Build Version: 2.17.50_86_g2f7fa4d [ 8174.389113] LNet: Added LNI 192.168.201.107@tcp [8/256/0/180] [ 8175.968168] Key type lgssc registered [ 8176.344181] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8178.914062] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8178.922054] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8180.085840] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8181.419303] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 8184.359263] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8184.486531] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8185.757502] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 8187.509216] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8188.681669] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8190.937066] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8191.059542] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8192.887975] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 8194.642861] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8206.439861] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 8211.938074] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8211.939311] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8211.942200] Lustre: Skipped 2 previous similar messages [ 8211.943779] Lustre: Skipped 1 previous similar message [ 8215.352287] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.7@tcp (stopping) [ 8215.354431] Lustre: Skipped 1 previous similar message [ 8217.057978] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8217.060911] Lustre: Skipped 2 previous similar messages [ 8217.148421] LustreError: 397817:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 8217.176082] Lustre: server umount lustre-MDT0000 complete [ 8217.949157] LustreError: 397170:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 192.168.201.7@tcp arrived at 1769763116 with bad export cookie 11781353319637680312 [ 8217.954087] LustreError: 397170:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8218.617287] LustreError: 397170:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769763116 with bad export cookie 11781353319637679661 [ 8218.621206] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8220.129053] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8220.131920] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8220.136684] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8222.177221] LustreError: 395555:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8222.181367] LustreError: 395555:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 8223.201166] LustreError: 395555:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8223.208238] LustreError: 395555:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8224.686091] LustreError: 398020:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8224.688029] LustreError: 398020:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8224.722822] Lustre: server umount lustre-OST0000 complete [ 8226.976939] LustreError: 398374:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 8226.978781] LustreError: 398374:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8227.038444] Lustre: server umount lustre-MDT0001 complete [ 8230.694842] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing unload_modules_local [ 8231.710219] Key type lgssc unregistered [ 8231.839399] LNet: 398904:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8231.841520] LNetError: 398904:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8231.851328] LNet: Removed LNI 192.168.201.107@tcp [ 8232.147836] Key type .llcrypt unregistered [ 8232.149501] Key type ._llcrypt unregistered [ 8240.923319] Key type ._llcrypt registered [ 8240.924602] Key type .llcrypt registered [ 8240.962327] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 8241.356285] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8241.388155] alg: No test for adler32 (adler32-zlib) [ 8242.237924] Lustre: Lustre: Build Version: 2.17.50_86_g2f7fa4d [ 8242.329685] LNet: Added LNI 192.168.201.107@tcp [8/256/0/180] [ 8243.904200] Key type lgssc registered [ 8244.273894] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8246.784207] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8246.791610] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8247.944734] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8249.174096] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 8251.942519] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8252.066665] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8253.325396] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 8255.040673] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8256.250262] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8258.413608] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8258.530589] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8260.310683] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 8262.064470] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8262.633095] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 8272.868384] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8272.868409] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 8272.873466] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8272.878041] Lustre: Skipped 1 previous similar message [ 8273.716183] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.7@tcp (stopping) [ 8275.334728] LustreError: 402519:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 8275.363877] Lustre: server umount lustre-MDT0001 complete [ 8277.986458] LustreError: 401699:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8277.986942] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8277.990974] LustreError: 401699:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8278.001554] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8278.009542] Lustre: Skipped 1 previous similar message [ 8278.012707] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8283.057126] LustreError: 402723:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8283.060114] LustreError: 402723:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8283.099213] Lustre: server umount lustre-OST0000 complete [ 8283.106268] LustreError: 400278:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8284.318391] LustreError: 402924:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 8284.320048] LustreError: 402924:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8284.370117] Lustre: server umount lustre-MDT0000 complete [ 8288.444361] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing unload_modules_local [ 8289.374305] Key type lgssc unregistered [ 8289.497400] LNet: 403606:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8289.499581] LNetError: 403606:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8289.511315] LNet: Removed LNI 192.168.201.107@tcp [ 8289.787128] Key type .llcrypt unregistered [ 8289.788099] Key type ._llcrypt unregistered [ 8297.818642] Key type ._llcrypt registered [ 8297.819649] Key type .llcrypt registered [ 8297.852017] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_hostid [ 8303.940183] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 8304.254324] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8304.260907] alg: No test for adler32 (adler32-zlib) [ 8305.113840] Lustre: Lustre: Build Version: 2.17.50_86_g2f7fa4d [ 8305.195489] LNet: Added LNI 192.168.201.107@tcp [8/256/0/180] [ 8306.776112] Key type lgssc registered [ 8307.115612] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8318.946344] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 8323.771700] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8323.776461] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8324.862822] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8324.872770] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8324.906939] Lustre: lustre-MDT0000: new disk, initializing [ 8324.930357] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8324.937530] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8326.207357] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 8331.736455] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8331.832371] Lustre: lustre-OST0000: new disk, initializing [ 8331.834309] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8331.836037] Lustre: Skipped 1 previous similar message [ 8331.858313] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8333.807215] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 8336.908482] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8337.406336] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 8337.409338] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 8337.417926] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 8347.487033] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8347.526393] Lustre: 407549:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8347.551917] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8347.586320] Lustre: lustre-MDT0001: new disk, initializing [ 8347.625294] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8347.640126] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 8347.644526] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 8348.880643] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 8352.369370] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 8352.447724] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 8353.823109] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 8353.906484] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 8357.859837] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8357.860852] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 8357.861084] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8357.864949] Lustre: Skipped 1 previous similar message [ 8361.535349] LustreError: 410955:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 8361.562291] Lustre: server umount lustre-MDT0001 complete [ 8368.098391] LustreError: 407555:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8368.102565] LustreError: 407555:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 8373.219418] LustreError: 410957:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8373.224306] LustreError: 410957:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8378.337664] LustreError: 410957:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8378.338119] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8378.338326] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8378.338938] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8378.338941] Lustre: Skipped 2 previous similar messages [ 8383.457736] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8383.458309] LustreError: 410957:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8388.064092] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8388.085965] LustreError: 411229:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 8388.087624] LustreError: 411229:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8388.119634] Lustre: server umount lustre-OST0000 complete [ 8388.577885] LustreError: 410957:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8395.516016] LustreError: 411431:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 8395.517725] LustreError: 411431:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8395.575422] Lustre: server umount lustre-MDT0000 complete [ 8399.938855] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing unload_modules_local [ 8400.943988] Key type lgssc unregistered [ 8401.073468] LNet: 412115:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8401.075890] LNetError: 412115:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8401.087374] LNet: Removed LNI 192.168.201.107@tcp [ 8401.399107] Key type .llcrypt unregistered [ 8401.400295] Key type ._llcrypt unregistered [ 8414.143300] Key type ._llcrypt registered [ 8414.144793] Key type .llcrypt registered [ 8414.182439] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing unload_modules_local [ 8414.271453] Key type .llcrypt unregistered [ 8414.272589] Key type ._llcrypt unregistered [ 8420.870541] Key type ._llcrypt registered [ 8420.871596] Key type .llcrypt registered [ 8420.904645] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 8421.202762] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8421.208660] alg: No test for adler32 (adler32-zlib) [ 8422.060505] Lustre: Lustre: Build Version: 2.17.50_86_g2f7fa4d [ 8422.137817] LNet: Added LNI 192.168.201.107@tcp [8/256/0/180] [ 8423.712097] Key type lgssc registered [ 8424.063029] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8432.555079] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8432.560506] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8433.646795] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8433.659366] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8433.693950] Lustre: lustre-MDT0000: new disk, initializing [ 8433.735383] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8433.742208] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8435.069280] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 8438.997200] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8439.035323] Lustre: 415425:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8439.058488] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8439.061275] Lustre: Skipped 1 previous similar message [ 8439.104910] Lustre: lustre-MDT0001: new disk, initializing [ 8439.135110] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8439.146366] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8439.149441] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8440.429527] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 8443.501186] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8444.738148] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8446.926805] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8447.039681] Lustre: lustre-OST0000: new disk, initializing [ 8447.041541] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8447.075962] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8448.925367] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 8450.487255] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8450.491086] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8450.522840] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8451.910789] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8454.624890] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8454.628042] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8454.633435] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8455.651081] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8455.651328] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8455.652812] Lustre: Skipped 1 previous similar message [ 8455.655925] Lustre: Skipped 1 previous similar message [ 8460.350570] LustreError: 417858:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 8460.384105] Lustre: server umount lustre-MDT0000 complete [ 8461.584085] LustreError: 416203:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769763359 with bad export cookie 10720449819939426940 [ 8461.589039] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8461.655896] LustreError: 418059:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8461.657964] LustreError: 418059:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8461.707908] Lustre: server umount lustre-MDT0001 complete [ 8473.141655] LustreError: 418261:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 8473.144082] LustreError: 418261:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8473.188640] Lustre: server umount lustre-OST0000 complete [ 8476.541897] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 03:56:14 (1769763374) [ 8478.925922] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8479.066678] LustreError: 419077:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8479.073132] LustreError: 419077:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 8479.090652] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8480.306113] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 8483.014990] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8483.123367] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8484.385381] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 8486.097407] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8487.316259] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8500.519401] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8500.597042] Lustre: Found index 0 for lustre-OST0000, updating log [ 8500.668204] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8502.427735] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 8504.079116] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8504.806906] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [ 8512.994563] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8512.995531] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8512.996748] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8513.004521] Lustre: Skipped 1 previous similar message [ 8516.334925] LustreError: 421332:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8516.336957] LustreError: 421332:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8516.371357] Lustre: server umount lustre-OST0000 complete [ 8529.376821] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8529.379234] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8529.383365] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8529.385038] Lustre: Skipped 3 previous similar messages [ 8535.521569] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8535.523781] Lustre: Skipped 3 previous similar messages [ 8543.200118] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8543.231375] LustreError: 421585:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 8543.233469] LustreError: 421585:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8543.258775] Lustre: server umount lustre-MDT0000 complete [ 8544.425379] LustreError: 419059:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769763442 with bad export cookie 10720449819939428319 [ 8544.427696] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8544.428655] LustreError: 419059:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 8544.544911] Lustre: server umount lustre-MDT0001 complete [ 8547.227157] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 03:57:25 (1769763445) [ 8558.368265] LNet: 1 local NIs in recovery (showing 1): 192.168.201.107@tcp [ 9126.669976] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9126.809622] LustreError: 425155:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9126.830954] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9128.116679] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 9131.201343] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9131.329943] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9132.772509] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 9134.636595] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9135.781718] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9148.994885] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9149.082180] Lustre: Found index 0 for lustre-OST0000, updating log [ 9149.187365] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9151.005104] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 9152.710788] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9153.319101] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:65) [ 9157.474613] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9157.474636] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9157.478318] Lustre: Skipped 1 previous similar message [ 9157.480863] Lustre: Skipped 5 previous similar messages [ 9159.468122] LustreError: 427458:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 9159.469919] LustreError: 427458:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 9159.505930] Lustre: server umount lustre-OST0000 complete [ 9161.762079] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9161.764107] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9161.765936] Lustre: Skipped 1 previous similar message [ 9163.809696] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9163.811360] Lustre: Skipped 4 previous similar messages [ 9167.905337] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9167.907181] Lustre: Skipped 7 previous similar messages [ 9175.008082] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9175.067174] Lustre: server umount lustre-MDT0000 complete [ 9175.074386] LustreError: 425173:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9176.097122] LustreError: 425173:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9176.103822] LustreError: 425173:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9176.274110] LustreError: 425134:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769764074 with bad export cookie 10720449819939429390 [ 9176.277823] LustreError: 425134:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9176.277994] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9176.342691] LustreError: 427870:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 9176.345221] LustreError: 427870:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 9176.395680] Lustre: server umount lustre-MDT0001 complete [ 9185.056258] LNet: 1 local NIs in recovery (showing 1): 192.168.201.107@tcp [ 9193.494805] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_hostid [ 9196.445150] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 9208.036835] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 04:08:26 (1769764106) [ 9213.674513] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing unload_modules_local [ 9214.772265] Key type lgssc unregistered [ 9214.907639] LNet: 431418:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9214.910207] LNetError: 431418:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9279.462319] LNet: Removed LNI 192.168.201.107@tcp [ 9279.734630] Key type .llcrypt unregistered [ 9279.735734] Key type ._llcrypt unregistered [ 9286.045264] Key type ._llcrypt registered [ 9286.046655] Key type .llcrypt registered [ 9286.085456] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 9286.388695] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9286.425280] alg: No test for adler32 (adler32-zlib) [ 9287.271640] Lustre: Lustre: Build Version: 2.17.50_86_g2f7fa4d [ 9287.353226] LNet: Added LNI 192.168.201.107@tcp [8/256/0/180] [ 9288.936183] Key type lgssc registered [ 9289.320760] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9290.650620] LNet: Added LNI 192.168.201.107@tcp100 [8/256/0/180] [ 9290.927820] LNet: Added LNI 192.168.201.107@tcp101 [8/256/0/180] [ 9291.200244] LNet: Added LNI 192.168.201.107@tcp102 [8/256/0/180] [ 9291.470603] LNet: Added LNI 192.168.201.107@tcp103 [8/256/0/180] [ 9291.753280] LNet: Added LNI 192.168.201.107@tcp104 [8/256/0/180] [ 9292.073097] LNet: Added LNI 192.168.201.107@tcp105 [8/256/0/180] [ 9292.384524] LNet: Added LNI 192.168.201.107@tcp106 [8/256/0/180] [ 9292.720200] LNet: Added LNI 192.168.201.107@tcp107 [8/256/0/180] [ 9293.058707] LNet: Added LNI 192.168.201.107@tcp108 [8/256/0/180] [ 9293.394516] LNet: Added LNI 192.168.201.107@tcp109 [8/256/0/180] [ 9293.715473] LNet: Added LNI 192.168.201.107@tcp110 [8/256/0/180] [ 9294.043926] LNet: Added LNI 192.168.201.107@tcp111 [8/256/0/180] [ 9294.337840] LNet: Added LNI 192.168.201.107@tcp112 [8/256/0/180] [ 9294.647823] LNet: Added LNI 192.168.201.107@tcp113 [8/256/0/180] [ 9294.953402] LNet: Added LNI 192.168.201.107@tcp114 [8/256/0/180] [ 9295.272539] LNet: Added LNI 192.168.201.107@tcp115 [8/256/0/180] [ 9295.572632] LNet: Added LNI 192.168.201.107@tcp116 [8/256/0/180] [ 9295.862950] LNet: Added LNI 192.168.201.107@tcp117 [8/256/0/180] [ 9296.158356] LNet: Added LNI 192.168.201.107@tcp118 [8/256/0/180] [ 9296.452753] LNet: Added LNI 192.168.201.107@tcp119 [8/256/0/180] [ 9296.789283] LNet: Added LNI 192.168.201.107@tcp120 [8/256/0/180] [ 9297.137412] LNet: Added LNI 192.168.201.107@tcp121 [8/256/0/180] [ 9297.491356] LNet: Added LNI 192.168.201.107@tcp122 [8/256/0/180] [ 9297.839675] LNet: Added LNI 192.168.201.107@tcp123 [8/256/0/180] [ 9298.146064] LNet: Added LNI 192.168.201.107@tcp124 [8/256/0/180] [ 9298.503618] LNet: Added LNI 192.168.201.107@tcp125 [8/256/0/180] [ 9298.819819] LNet: Added LNI 192.168.201.107@tcp126 [8/256/0/180] [ 9299.209382] LNet: Added LNI 192.168.201.107@tcp127 [8/256/0/180] [ 9299.548078] LNet: Added LNI 192.168.201.107@tcp128 [8/256/0/180] [ 9299.824627] LNet: Added LNI 192.168.201.107@tcp129 [8/256/0/180] [ 9300.109183] LNet: Added LNI 192.168.201.107@tcp130 [8/256/0/180] [ 9300.389571] LNet: Added LNI 192.168.201.107@tcp131 [8/256/0/180] [ 9300.667492] LNet: Added LNI 192.168.201.107@tcp132 [8/256/0/180] [ 9300.970907] LNet: Added LNI 192.168.201.107@tcp133 [8/256/0/180] [ 9301.298662] LNet: Added LNI 192.168.201.107@tcp134 [8/256/0/180] [ 9301.626366] LNet: Added LNI 192.168.201.107@tcp135 [8/256/0/180] [ 9304.034945] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9304.042169] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9305.102283] Lustre: 434395:0:(lustre_peer.c:105:class_add_uuid()) 192.168.201.107@tcp: can't add NID 192.168.201.107@tcp130: rc = -75 [ 9305.192300] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9305.204111] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9305.239419] Lustre: lustre-MDT0000: new disk, initializing [ 9305.272526] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9305.279069] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9306.145204] LNetError: 434419:0:(socklnd_cb.c:1815:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.107 [ 9306.148613] LNetError: Protocol error connecting to 192.168.201.107@tcp108 at host 192.168.201.107:988: is it running a compatible version of Lustre? [ 9306.608549] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 9310.862753] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9310.915217] Lustre: 434413:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9310.949085] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9310.951923] Lustre: Skipped 1 previous similar message [ 9310.999943] Lustre: lustre-MDT0001: new disk, initializing [ 9311.013528] Lustre: 435197:0:(lustre_peer.c:105:class_add_uuid()) 192.168.201.107@tcp: can't add NID 192.168.201.107@tcp130: rc = -75 [ 9311.017426] Lustre: 435197:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [ 9311.069669] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9311.083874] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9311.088447] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9312.450920] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 9315.783793] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9316.849624] Lustre: 435771:0:(lustre_peer.c:105:class_add_uuid()) 192.168.201.107@tcp: can't add NID 192.168.201.107@tcp130: rc = -75 [ 9316.852325] Lustre: 435771:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [ 9316.917966] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9319.221960] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9319.344529] Lustre: lustre-OST0000: new disk, initializing [ 9319.346754] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9319.381525] Lustre: 436166:0:(lustre_peer.c:105:class_add_uuid()) 192.168.201.107@tcp: can't add NID 192.168.201.107@tcp130: rc = -75 [ 9319.387350] Lustre: 436166:0:(lustre_peer.c:105:class_add_uuid()) Skipped 5 previous similar messages [ 9319.408484] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9321.154902] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 9323.080000] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9323.084190] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9323.125192] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9324.152001] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9328.096858] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9328.100822] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9328.105744] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9328.612083] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9328.615429] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9328.620963] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9332.461105] LustreError: 436997:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 9332.496567] Lustre: server umount lustre-OST0000 complete [ 9336.801237] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9336.804774] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9336.810400] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9338.849698] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9338.850668] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9341.665847] LNetError: 431862:0:(socklnd_cb.c:1815:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.107 [ 9341.671787] LNetError: Protocol error connecting to 192.168.201.107@tcp108 at host 192.168.201.107:988: is it running a compatible version of Lustre? [ 9343.969375] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9343.971312] Lustre: Skipped 2 previous similar messages [ 9348.064113] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9348.108829] LustreError: 437200:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 9348.112431] LustreError: 437200:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 9348.147797] Lustre: server umount lustre-MDT0000 complete [ 9349.419209] LustreError: 435196:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769764247 with bad export cookie 6533965622982445985 [ 9349.423251] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9349.424586] LustreError: 435196:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9349.502852] LustreError: 437402:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 9349.505174] LustreError: 437402:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 9349.562305] Lustre: server umount lustre-MDT0001 complete [ 9352.973947] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing unload_modules_local [ 9354.115347] Key type lgssc unregistered [ 9354.289223] LNet: 437941:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9354.293800] LNetError: 437941:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9354.297988] LNet: Removed LNI 192.168.201.107@tcp [ 9355.360169] LNet: Removed LNI 192.168.201.107@tcp100 [ 9355.362248] LNetError: 437941:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9355.365562] LNetError: 437941:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 9355.369255] LNet: Removed LNI 192.168.201.107@tcp101 [ 9356.384178] LNet: Removed LNI 192.168.201.107@tcp102 [ 9356.385601] LNetError: 437941:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9356.387818] LNetError: 437941:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 9356.390307] LNet: Removed LNI 192.168.201.107@tcp103 [ 9357.408147] LNet: Removed LNI 192.168.201.107@tcp104 [ 9358.432148] LNet: Removed LNI 192.168.201.107@tcp105 [ 9358.434107] LNetError: 437941:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9358.436206] LNetError: 437941:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 9358.438394] LNet: Removed LNI 192.168.201.107@tcp106 [ 9359.456112] LNet: Removed LNI 192.168.201.107@tcp107 [ 9360.480102] LNet: Removed LNI 192.168.201.107@tcp108 [ 9361.504090] LNet: Removed LNI 192.168.201.107@tcp109 [ 9362.528104] LNet: Removed LNI 192.168.201.107@tcp110 [ 9362.529763] LNetError: 437941:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9362.532198] LNetError: 437941:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 9362.534571] LNet: Removed LNI 192.168.201.107@tcp111 [ 9363.552083] LNet: Removed LNI 192.168.201.107@tcp112 [ 9364.576098] LNet: Removed LNI 192.168.201.107@tcp113 [ 9365.600085] LNet: Removed LNI 192.168.201.107@tcp114 [ 9366.624092] LNet: Removed LNI 192.168.201.107@tcp115 [ 9367.648083] LNet: Removed LNI 192.168.201.107@tcp116 [ 9367.649396] LNet: Removed LNI 192.168.201.107@tcp117 [ 9367.650524] LNet: Removed LNI 192.168.201.107@tcp118 [ 9367.651675] LNet: Removed LNI 192.168.201.107@tcp119 [ 9367.652764] LNet: Removed LNI 192.168.201.107@tcp120 [ 9367.653872] LNet: Removed LNI 192.168.201.107@tcp121 [ 9367.655014] LNet: Removed LNI 192.168.201.107@tcp122 [ 9367.656077] LNet: Removed LNI 192.168.201.107@tcp123 [ 9367.657163] LNet: Removed LNI 192.168.201.107@tcp124 [ 9367.658301] LNet: Removed LNI 192.168.201.107@tcp125 [ 9367.659940] LNet: Removed LNI 192.168.201.107@tcp126 [ 9367.661065] LNet: Removed LNI 192.168.201.107@tcp127 [ 9367.662213] LNet: Removed LNI 192.168.201.107@tcp128 [ 9367.663404] LNet: Removed LNI 192.168.201.107@tcp129 [ 9367.664611] LNet: Removed LNI 192.168.201.107@tcp130 [ 9367.665735] LNet: Removed LNI 192.168.201.107@tcp131 [ 9367.666970] LNet: Removed LNI 192.168.201.107@tcp132 [ 9367.668159] LNet: Removed LNI 192.168.201.107@tcp133 [ 9367.669318] LNet: Removed LNI 192.168.201.107@tcp134 [ 9367.676413] LNet: Removed LNI 192.168.201.107@tcp135 [ 9367.960717] Key type .llcrypt unregistered [ 9367.961665] Key type ._llcrypt unregistered [ 9375.574270] Key type ._llcrypt registered [ 9375.576087] Key type .llcrypt registered [ 9375.615789] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_hostid [ 9381.654738] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 9382.021628] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9382.082518] alg: No test for adler32 (adler32-zlib) [ 9382.944077] Lustre: Lustre: Build Version: 2.17.50_86_g2f7fa4d [ 9383.033749] LNet: Added LNI 192.168.201.107@tcp [8/256/0/180] [ 9384.616125] Key type lgssc registered [ 9384.967067] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9396.415361] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 04:11:34 (1769764294) [ 9401.909928] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing unload_modules_local [ 9402.929702] Key type lgssc unregistered [ 9403.054359] LNet: 441943:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9403.057858] LNetError: 441943:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9403.067271] LNet: Removed LNI 192.168.201.107@tcp [ 9403.334443] Key type .llcrypt unregistered [ 9403.335710] Key type ._llcrypt unregistered [ 9409.232605] Key type ._llcrypt registered [ 9409.233608] Key type .llcrypt registered [ 9409.267396] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 9409.577193] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9409.609366] alg: No test for adler32 (adler32-zlib) [ 9410.447762] Lustre: Lustre: Build Version: 2.17.50_86_g2f7fa4d [ 9410.524142] LNet: Added LNI 192.168.201.107@tcp [8/256/0/180] [ 9412.112121] Key type lgssc registered [ 9412.466683] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9415.370489] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9415.375670] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9416.469202] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9416.482777] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9416.522984] Lustre: lustre-MDT0000: new disk, initializing [ 9416.546350] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9416.552344] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9417.755665] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 9421.771023] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9421.809755] Lustre: 443112:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9421.829880] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9421.831471] Lustre: Skipped 1 previous similar message [ 9421.865420] Lustre: lustre-MDT0001: new disk, initializing [ 9421.894986] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9421.905512] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9421.908390] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9423.019273] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 9425.961677] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9427.233910] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9429.382466] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9429.481986] Lustre: lustre-OST0000: new disk, initializing [ 9429.483880] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9429.512456] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9430.465436] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9430.469746] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9430.498501] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9431.334219] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 9434.256713] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9435.739248] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing lnet_if_list [ 9437.145858] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [ 9437.162248] LNet: Added LNI 192.168.201.107@tcp10 [8/256/0/180] [ 9438.848995] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [ 9438.859753] LNet: Added LNI 192.168.201.107@tcp11 [8/256/0/180] [ 9440.201158] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [ 9440.211920] LNet: Added LNI 192.168.201.107@tcp12 [8/256/0/180] [ 9441.535997] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [ 9441.544785] LNet: Added LNI 192.168.201.107@tcp13 [8/256/0/180] [ 9442.891407] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [ 9442.904696] LNet: Added LNI 192.168.201.107@tcp14 [8/256/0/180] [ 9444.085182] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [ 9444.093584] LNet: Added LNI 192.168.201.107@tcp15 [8/256/0/180] [ 9445.458628] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [ 9445.473972] LNet: Added LNI 192.168.201.107@tcp16 [8/256/0/180] [ 9446.836366] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [ 9446.846388] LNet: Added LNI 192.168.201.107@tcp17 [8/256/0/180] [ 9448.133098] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [ 9448.141685] LNet: Added LNI 192.168.201.107@tcp18 [8/256/0/180] [ 9449.408595] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [ 9449.419636] LNet: Added LNI 192.168.201.107@tcp19 [8/256/0/180] [ 9451.052321] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [ 9451.061368] LNet: Added LNI 192.168.201.107@tcp20 [8/256/0/180] [ 9452.466694] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [ 9452.483388] LNet: Added LNI 192.168.201.107@tcp21 [8/256/0/180] [ 9454.131733] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [ 9454.140268] LNet: Added LNI 192.168.201.107@tcp22 [8/256/0/180] [ 9455.702100] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [ 9455.712979] LNet: Added LNI 192.168.201.107@tcp23 [8/256/0/180] [ 9457.207951] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [ 9457.216519] LNet: Added LNI 192.168.201.107@tcp24 [8/256/0/180] [ 9458.878204] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [ 9458.888489] LNet: Added LNI 192.168.201.107@tcp25 [8/256/0/180] [ 9460.489563] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [ 9460.501381] LNet: Added LNI 192.168.201.107@tcp26 [8/256/0/180] [ 9462.358214] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [ 9462.370233] LNet: Added LNI 192.168.201.107@tcp27 [8/256/0/180] [ 9464.184579] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [ 9464.196463] LNet: Added LNI 192.168.201.107@tcp28 [8/256/0/180] [ 9465.959224] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [ 9465.969171] LNet: Added LNI 192.168.201.107@tcp29 [8/256/0/180] [ 9467.776426] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [ 9467.792369] LNet: Added LNI 192.168.201.107@tcp30 [8/256/0/180] [ 9469.924894] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [ 9469.935327] LNet: Added LNI 192.168.201.107@tcp31 [8/256/0/180] [ 9471.814133] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [ 9471.823782] LNet: Added LNI 192.168.201.107@tcp32 [8/256/0/180] [ 9473.666753] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [ 9473.679412] LNet: Added LNI 192.168.201.107@tcp33 [8/256/0/180] [ 9475.558759] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [ 9475.567726] LNet: Added LNI 192.168.201.107@tcp34 [8/256/0/180] [ 9477.452157] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [ 9477.460635] LNet: Added LNI 192.168.201.107@tcp35 [8/256/0/180] [ 9479.481333] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [ 9479.490201] LNet: Added LNI 192.168.201.107@tcp36 [8/256/0/180] [ 9481.411286] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [ 9481.421326] LNet: Added LNI 192.168.201.107@tcp37 [8/256/0/180] [ 9483.398803] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [ 9483.409316] LNet: Added LNI 192.168.201.107@tcp38 [8/256/0/180] [ 9485.507467] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [ 9485.516758] LNet: Added LNI 192.168.201.107@tcp39 [8/256/0/180] [ 9487.671369] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [ 9487.682498] LNet: Added LNI 192.168.201.107@tcp40 [8/256/0/180] [ 9489.940409] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [ 9489.949508] LNet: Added LNI 192.168.201.107@tcp41 [8/256/0/180] [ 9492.241155] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [ 9492.251524] LNet: Added LNI 192.168.201.107@tcp42 [8/256/0/180] [ 9494.426240] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [ 9494.435374] LNet: Added LNI 192.168.201.107@tcp43 [8/256/0/180] [ 9496.633666] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [ 9496.646703] LNet: Added LNI 192.168.201.107@tcp44 [8/256/0/180] [ 9499.001359] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [ 9499.012818] LNet: Added LNI 192.168.201.107@tcp45 [8/256/0/180] [ 9501.453850] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [ 9501.464646] LNet: Added LNI 192.168.201.107@tcp46 [8/256/0/180] [ 9503.643158] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [ 9503.651713] LNet: Added LNI 192.168.201.107@tcp47 [8/256/0/180] [ 9505.845400] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [ 9505.855914] LNet: Added LNI 192.168.201.107@tcp48 [8/256/0/180] [ 9508.224931] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [ 9508.235893] LNet: Added LNI 192.168.201.107@tcp49 [8/256/0/180] [ 9510.583586] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [ 9510.596714] LNet: Added LNI 192.168.201.107@tcp50 [8/256/0/180] [ 9519.073252] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9519.077453] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9519.084693] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9522.658347] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9522.660517] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9522.664282] Lustre: Skipped 1 previous similar message [ 9523.974074] LustreError: 467138:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 9524.021671] Lustre: server umount lustre-MDT0000 complete [ 9525.210267] LustreError: 443891:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769764423 with bad export cookie 15669096679281801573 [ 9525.213076] LustreError: 443891:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9525.214445] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9525.274936] LustreError: 467340:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 9525.276515] LustreError: 467340:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 9525.325032] Lustre: server umount lustre-MDT0001 complete [ 9532.457647] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 9535.342500] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9543.136080] Lustre: 442388:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1769764425/real 1769764425] req@00000000098c8faa x1855732379104384/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1769764441 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9543.142523] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9546.144188] Lustre: 442385:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1769764428/real 1769764428] req@000000004379f7b0 x1855732379104512/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1769764444 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9547.936129] Lustre: 442386:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1769764430/real 1769764430] req@00000000184885a1 x1855732379104640/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1769764446 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9550.304823] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xd973d0694fd84130 [ 9550.307238] Lustre: MGC192.168.201.107@tcp: Connection restored to 0@lo (at 0@lo) [ 9550.413537] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9551.456132] Lustre: 442387:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1769764433/real 1769764433] req@00000000da275fa2 x1855732379104896/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1769764449 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9551.638126] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 9554.535917] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9554.713599] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9556.161931] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 9556.448121] Lustre: 442388:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1769764438/real 1769764438] req@00000000861aa261 x1855732379105024/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1769764454 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9558.441343] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9559.988744] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9560.039628] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9562.085772] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9570.274959] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9570.280804] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9570.290929] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9570.295426] Lustre: Skipped 2 previous similar messages [ 9572.323261] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9572.324840] Lustre: Skipped 2 previous similar messages [ 9574.391363] LustreError: 469829:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 9574.394960] LustreError: 469829:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 9574.463703] Lustre: server umount lustre-OST0000 complete [ 9577.442550] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9577.445711] Lustre: Skipped 1 previous similar message [ 9577.447389] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9577.450298] Lustre: Skipped 1 previous similar message [ 9587.682209] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9587.685014] Lustre: Skipped 3 previous similar messages [ 9590.240119] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9590.295498] LustreError: 470031:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 9590.297210] LustreError: 470031:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 9590.325864] Lustre: server umount lustre-MDT0000 complete [ 9591.820258] LustreError: 443103:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769764489 with bad export cookie 15669096679281803568 [ 9591.823305] LustreError: 443103:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9591.823388] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9591.932528] Lustre: server umount lustre-MDT0001 complete [ 9595.477656] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing unload_modules_local [ 9596.706377] Key type lgssc unregistered [ 9596.847851] LNet: 470772:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9596.850357] LNetError: 470772:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9596.852359] LNet: Removed LNI 192.168.201.107@tcp [ 9596.853354] LNet: Removed LNI 192.168.201.107@tcp10 [ 9596.854366] LNet: Removed LNI 192.168.201.107@tcp11 [ 9596.855408] LNet: Removed LNI 192.168.201.107@tcp12 [ 9596.856435] LNet: Removed LNI 192.168.201.107@tcp13 [ 9596.857408] LNet: Removed LNI 192.168.201.107@tcp14 [ 9596.858437] LNet: Removed LNI 192.168.201.107@tcp15 [ 9596.859452] LNet: Removed LNI 192.168.201.107@tcp16 [ 9596.860480] LNet: Removed LNI 192.168.201.107@tcp17 [ 9596.861449] LNet: Removed LNI 192.168.201.107@tcp18 [ 9596.862479] LNet: Removed LNI 192.168.201.107@tcp19 [ 9596.863469] LNet: Removed LNI 192.168.201.107@tcp20 [ 9596.864489] LNet: Removed LNI 192.168.201.107@tcp21 [ 9596.865524] LNet: Removed LNI 192.168.201.107@tcp22 [ 9596.866512] LNet: Removed LNI 192.168.201.107@tcp23 [ 9596.867547] LNet: Removed LNI 192.168.201.107@tcp24 [ 9596.868543] LNet: Removed LNI 192.168.201.107@tcp25 [ 9596.869566] LNet: Removed LNI 192.168.201.107@tcp26 [ 9596.870682] LNet: Removed LNI 192.168.201.107@tcp27 [ 9596.871681] LNet: Removed LNI 192.168.201.107@tcp28 [ 9596.872707] LNet: Removed LNI 192.168.201.107@tcp29 [ 9596.873679] LNet: Removed LNI 192.168.201.107@tcp30 [ 9596.874763] LNet: Removed LNI 192.168.201.107@tcp31 [ 9596.875768] LNet: Removed LNI 192.168.201.107@tcp32 [ 9596.876762] LNet: Removed LNI 192.168.201.107@tcp33 [ 9596.877750] LNet: Removed LNI 192.168.201.107@tcp34 [ 9596.878745] LNet: Removed LNI 192.168.201.107@tcp35 [ 9596.879751] LNet: Removed LNI 192.168.201.107@tcp36 [ 9596.880830] LNet: Removed LNI 192.168.201.107@tcp37 [ 9596.881861] LNet: Removed LNI 192.168.201.107@tcp38 [ 9596.882855] LNet: Removed LNI 192.168.201.107@tcp39 [ 9596.883875] LNet: Removed LNI 192.168.201.107@tcp40 [ 9596.884895] LNet: Removed LNI 192.168.201.107@tcp41 [ 9596.885882] LNet: Removed LNI 192.168.201.107@tcp42 [ 9596.886869] LNet: Removed LNI 192.168.201.107@tcp43 [ 9596.887828] LNet: Removed LNI 192.168.201.107@tcp44 [ 9596.888797] LNet: Removed LNI 192.168.201.107@tcp45 [ 9596.889819] LNet: Removed LNI 192.168.201.107@tcp46 [ 9596.890809] LNet: Removed LNI 192.168.201.107@tcp47 [ 9596.891842] LNet: Removed LNI 192.168.201.107@tcp48 [ 9596.892805] LNet: Removed LNI 192.168.201.107@tcp49 [ 9596.899314] LNet: Removed LNI 192.168.201.107@tcp50 [ 9597.127419] Key type .llcrypt unregistered [ 9597.128492] Key type ._llcrypt unregistered [ 9605.731606] Key type ._llcrypt registered [ 9605.733202] Key type .llcrypt registered [ 9605.770261] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 9606.113712] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9606.134106] alg: No test for adler32 (adler32-zlib) [ 9606.976187] Lustre: Lustre: Build Version: 2.17.50_86_g2f7fa4d [ 9607.055148] LNet: Added LNI 192.168.201.107@tcp [8/256/0/180] [ 9608.632131] Key type lgssc registered [ 9609.044471] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9611.444276] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9611.452310] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9612.604386] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9613.859578] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 9616.787171] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9616.900840] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9618.035555] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 9619.981936] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9621.074055] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9623.731372] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9623.864794] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9625.891814] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 9627.871640] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9638.193640] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 9641.895056] loop0: detected capacity change from 0 to 204800000 [ 9641.955219] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9641.957932] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9641.961140] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9644.031473] loop0: detected capacity change from 0 to 204800000 [ 9644.149830] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9644.153030] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9644.156487] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9645.849610] loop0: detected capacity change from 0 to 204800000 [ 9647.779973] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9647.829320] Lustre: fs15246-OST0000: new disk, initializing [ 9647.831136] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [ 9647.847427] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9649.656463] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 9653.331922] loop1: detected capacity change from 0 to 204800000 [ 9655.185280] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9655.221970] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [ 9655.246936] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [ 9655.285940] Lustre: fs15246-MDT0000: new disk, initializing [ 9655.342918] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9655.350968] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9655.387488] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 9656.416994] Lustre: 476375:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@00000000b125e603 x1855732585627136/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1769764570 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 9656.586730] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 9665.504419] LustreError: 476375:0:(fid_request.c:213:seq_client_alloc_seq()) cli-cli-fs15246-OST0000-osc-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 9665.507367] LustreError: 476375:0:(fid_request.c:255:seq_client_get_seq()) cli-cli-fs15246-OST0000-osc-MDT0000: Can't allocate new sequence: rc = -5 [ 9665.509956] LustreError: 476375:0:(osp_precreate.c:459:osp_precreate_rollover_new_seq()) fs15246-OST0000-osc-MDT0000: alloc fid error: rc = -5 [ 9665.518123] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [ 9669.439805] LustreError: 477492:0:(obd_class.h:479:obd_check_dev()) Device 30 not setup [ 9669.467633] Lustre: server umount fs15246-MDT0000 complete [ 9670.627924] LustreError: 474145:0:(ldlm_lib.c:1179:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9670.632693] LustreError: 474145:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9672.347475] LustreError: 477996:0:(obd_class.h:479:obd_check_dev()) Device 29 not setup [ 9672.349116] LustreError: 477996:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 9672.361237] Lustre: server umount fs15246-OST0000 complete [ 9675.745590] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9675.747732] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9675.748805] Lustre: Skipped 1 previous similar message [ 9680.868860] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9680.872075] Lustre: Skipped 2 previous similar messages [ 9681.581096] LustreError: 478497:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 9681.582814] LustreError: 478497:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 9681.615317] Lustre: server umount lustre-OST0000 complete [ 9683.424991] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9683.427184] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9683.430576] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9685.986132] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9691.105424] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9691.109850] Lustre: Skipped 3 previous similar messages [ 9697.248135] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9697.287132] LustreError: 478699:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 9697.288672] LustreError: 478699:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 9697.313508] Lustre: server umount lustre-MDT0000 complete [ 9698.426322] LustreError: 471972:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769764596 with bad export cookie 76935237593291453 [ 9698.429197] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9698.432406] LustreError: 471972:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9698.529307] Lustre: server umount lustre-MDT0001 complete [ 9701.784829] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing unload_modules_local [ 9702.752224] Key type lgssc unregistered [ 9702.865297] LNet: 479431:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9702.867327] LNetError: 479431:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9702.876285] LNet: Removed LNI 192.168.201.107@tcp [ 9703.177924] Key type .llcrypt unregistered [ 9703.179330] Key type ._llcrypt unregistered [ 9710.017144] Key type ._llcrypt registered [ 9710.018166] Key type .llcrypt registered [ 9710.052975] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 9710.367541] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9710.398073] alg: No test for adler32 (adler32-zlib) [ 9711.240931] Lustre: Lustre: Build Version: 2.17.50_86_g2f7fa4d [ 9711.320547] LNet: Added LNI 192.168.201.107@tcp [8/256/0/180] [ 9712.896112] Key type lgssc registered [ 9713.287785] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9725.392565] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_hostid [ 9728.171188] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 9739.005961] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9739.010938] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9740.100275] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9740.110045] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9740.146727] Lustre: lustre-MDT0000: new disk, initializing [ 9740.171193] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9740.177740] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9741.379401] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 9745.359245] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9745.397911] Lustre: 483812:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9745.421530] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9745.423020] Lustre: Skipped 1 previous similar message [ 9745.457789] Lustre: lustre-MDT0001: new disk, initializing [ 9745.487911] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9745.499990] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9745.502792] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9746.707818] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 9749.678816] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9750.804665] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9752.962820] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9753.086986] Lustre: lustre-OST0000: new disk, initializing [ 9753.089556] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9753.125275] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9754.904180] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 9757.803625] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9758.201303] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9758.205523] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9758.236600] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9759.023071] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9759.090604] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9760.295329] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9760.353702] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9763.298258] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9763.300552] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9763.302521] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9763.308978] Lustre: Skipped 1 previous similar message [ 9766.957875] LustreError: 486745:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 9766.990050] Lustre: server umount lustre-OST0000 complete [ 9768.417285] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9768.418036] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9768.420636] Lustre: Skipped 1 previous similar message [ 9768.423216] Lustre: Skipped 1 previous similar message [ 9773.537168] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9773.540352] Lustre: Skipped 1 previous similar message [ 9774.271811] LustreError: 486946:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 9774.274045] LustreError: 486946:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 9774.299443] Lustre: server umount lustre-MDT0000 complete [ 9775.500369] LustreError: 483804:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769764673 with bad export cookie 12716578781352403378 [ 9775.504141] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9775.506222] LustreError: 483804:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9775.576723] LustreError: 487147:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 9775.578487] LustreError: 487147:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 9775.625398] Lustre: server umount lustre-MDT0001 complete [ 9783.345137] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_hostid [ 9786.686282] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 9797.482961] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 9461 sec ========== 04:18:15 (1769764695) [ 9798.043202] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 04:18:16 (1769764696) === [ 9799.220626] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 04:18:17 (1769764697) === [ 9811.275250] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing unload_modules_local [ 9812.169689] Key type lgssc unregistered [ 9812.284234] LNet: 491985:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9812.286194] LNetError: 491985:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9812.292283] LNet: Removed LNI 192.168.201.107@tcp [ 9812.557628] Key type .llcrypt unregistered [ 9812.558570] Key type ._llcrypt unregistered