[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 460564909 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2439 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22D5 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 002295 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2349 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23D9 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2411 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22d5-0xbffe2348] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22d4] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2349-0xbffe23d8] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23d9-0xbffe2410] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2411-0xbffe2438] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003149] x2apic enabled [ 0.004006] Switched APIC routing to physical x2apic. [ 0.005010] kvm-guest: setup PV IPIs [ 0.008308] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009020] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010014] pid_max: default: 32768 minimum: 301 [ 0.011131] LSM: Security Framework initializing [ 0.012073] Yama: becoming mindful. [ 0.013036] SELinux: Initializing. [ 0.015059] *** VALIDATE selinux *** [ 0.022762] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026767] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027158] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029073] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030123] *** VALIDATE tmpfs *** [ 0.031429] *** VALIDATE proc *** [ 0.032281] *** VALIDATE cgroup *** [ 0.033010] *** VALIDATE cgroup2 *** [ 0.035180] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036158] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037011] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038031] Spectre V2 : User space: Vulnerable [ 0.039011] Speculative Store Bypass: Vulnerable [ 0.041864] debug: unmapping init [mem 0xffffffff8e659000-0xffffffff8e660fff] [ 0.043192] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044656] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045023] ... version: 2 [ 0.046010] ... bit width: 48 [ 0.046993] ... generic registers: 4 [ 0.047011] ... value mask: 0000ffffffffffff [ 0.048025] ... max period: 00007fffffffffff [ 0.049011] ... fixed-purpose events: 3 [ 0.050012] ... event mask: 000000070000000f [ 0.051270] rcu: Hierarchical SRCU implementation. [ 0.053398] smp: Bringing up secondary CPUs ... [ 0.054558] x86: Booting SMP configuration: [ 0.055022] .... node #0, CPUs: #1 #2 #3 [ 0.058341] smp: Brought up 1 node, 4 CPUs [ 0.060012] smpboot: Max logical packages: 1 [ 0.061014] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.143020] node 0 deferred pages initialised in 80ms [ 0.145303] devtmpfs: initialized [ 0.147277] x86/mm: Memory block size: 128MB [ 0.150299] gcov: version magic: 0x41383552 [ 0.153288] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.154078] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.155264] pinctrl core: initialized pinctrl subsystem [ 0.156188] [ 0.156801] ************************************************************* [ 0.157015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.158011] ** ** [ 0.159015] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.160015] ** ** [ 0.161013] ** This means that this kernel is built to expose internal ** [ 0.162012] ** IOMMU data structures, which may compromise security on ** [ 0.163016] ** your system. ** [ 0.164015] ** ** [ 0.165014] ** If you see this message and you are not debugging the ** [ 0.166012] ** kernel, report this immediately to your vendor! ** [ 0.167016] ** ** [ 0.168016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.169012] ************************************************************* [ 0.170662] NET: Registered protocol family 16 [ 0.171432] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.172054] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.173078] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.174440] cpuidle: using governor menu [ 0.176850] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.180395] PCI: Using configuration type 1 for base access [ 0.183114] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.195071] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.196024] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.198138] cryptd: max_cpu_qlen set to 1000 [ 0.202074] ACPI: Added _OSI(Module Device) [ 0.203025] ACPI: Added _OSI(Processor Device) [ 0.206015] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.208017] ACPI: Added _OSI(Processor Aggregator Device) [ 0.213831] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.220315] ACPI: Interpreter enabled [ 0.222052] ACPI: PM: (supports S0 S3 S4 S5) [ 0.223011] ACPI: Using IOAPIC for interrupt routing [ 0.224118] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.227299] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.235299] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.237038] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.240019] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.243072] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.246930] acpiphp: Slot [2] registered [ 0.248090] acpiphp: Slot [5] registered [ 0.249091] acpiphp: Slot [6] registered [ 0.251111] acpiphp: Slot [7] registered [ 0.252096] acpiphp: Slot [8] registered [ 0.253099] acpiphp: Slot [9] registered [ 0.254085] acpiphp: Slot [10] registered [ 0.255095] acpiphp: Slot [3] registered [ 0.256136] acpiphp: Slot [4] registered [ 0.258114] acpiphp: Slot [11] registered [ 0.259098] acpiphp: Slot [12] registered [ 0.260070] acpiphp: Slot [13] registered [ 0.262115] acpiphp: Slot [14] registered [ 0.263099] acpiphp: Slot [15] registered [ 0.265096] acpiphp: Slot [16] registered [ 0.267095] acpiphp: Slot [17] registered [ 0.268094] acpiphp: Slot [18] registered [ 0.270088] acpiphp: Slot [19] registered [ 0.271092] acpiphp: Slot [20] registered [ 0.272031] acpiphp: Slot [21] registered [ 0.272794] acpiphp: Slot [22] registered [ 0.274045] acpiphp: Slot [23] registered [ 0.274702] acpiphp: Slot [24] registered [ 0.275048] acpiphp: Slot [25] registered [ 0.275867] acpiphp: Slot [26] registered [ 0.277077] acpiphp: Slot [27] registered [ 0.279062] acpiphp: Slot [28] registered [ 0.280070] acpiphp: Slot [29] registered [ 0.281110] acpiphp: Slot [30] registered [ 0.283093] acpiphp: Slot [31] registered [ 0.284053] PCI host bridge to bus 0000:00 [ 0.285017] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.287022] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.289022] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.292099] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.294025] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.297025] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.299176] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.301838] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.305018] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.312015] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.317348] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.320019] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.322018] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.324019] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.327595] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.330785] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.334047] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.337872] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.342013] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.355014] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.360015] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.365743] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.373013] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.380016] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.406015] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.415181] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.425014] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.432017] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.458017] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.468457] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.477016] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.490019] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.515024] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.528722] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.536018] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.545019] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.569019] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.579907] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.590016] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.596016] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.622014] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.632833] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.641015] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.654019] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.683015] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.694448] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.696390] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.699361] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.702430] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.704236] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.709604] iommu: Default domain type: Passthrough [ 0.711455] SCSI subsystem initialized [ 0.713130] ACPI: bus type USB registered [ 0.715097] usbcore: registered new interface driver usbfs [ 0.717090] usbcore: registered new interface driver hub [ 0.719081] usbcore: registered new device driver usb [ 0.721176] pps_core: LinuxPPS API ver. 1 registered [ 0.723011] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.726057] PTP clock support registered [ 0.728141] EDAC MC: Ver: 3.0.0 [ 0.730110] PCI: Using ACPI for IRQ routing [ 0.731809] NetLabel: Initializing [ 0.734012] NetLabel: domain hash size = 128 [ 0.735010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.737083] NetLabel: unlabeled traffic allowed by default [ 0.740084] vgaarb: loaded [ 0.742158] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.744015] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.751852] clocksource: Switched to clocksource kvm-clock [ 0.856244] VFS: Disk quotas dquot_6.6.0 [ 0.857903] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.860868] *** VALIDATE ramfs *** [ 0.862179] *** VALIDATE hugetlbfs *** [ 0.863744] pnp: PnP ACPI init [ 0.865890] pnp: PnP ACPI: found 6 devices [ 0.885099] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.887766] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.890173] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.892312] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.894645] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.896991] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.899827] NET: Registered protocol family 2 [ 0.902225] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.906946] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.910575] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.915836] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.919591] TCP: Hash tables configured (established 65536 bind 65536) [ 0.922546] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.924861] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.927057] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.929844] NET: Registered protocol family 1 [ 0.931455] RPC: Registered named UNIX socket transport module. [ 0.932913] RPC: Registered udp transport module. [ 0.933989] RPC: Registered tcp transport module. [ 0.935187] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.937443] NET: Registered protocol family 44 [ 0.939196] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.940665] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.942147] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.943748] PCI: CLS 0 bytes, default 64 [ 0.944837] Unpacking initramfs... [ 2.320713] debug: unmapping init [mem 0xffff94a8fcc54000-0xffff94a8fffbffff] [ 2.325047] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.327503] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.330692] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.846224] Initialise system trusted keyrings [ 2.847966] Key type blacklist registered [ 2.849744] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.857390] zbud: loaded [ 2.859981] *** VALIDATE nfs *** [ 2.861137] *** VALIDATE nfs4 *** [ 2.862450] pstore: using deflate compression [ 2.865393] Platform Keyring initialized [ 2.947756] NET: Registered protocol family 38 [ 2.949258] Key type asymmetric registered [ 2.950804] Asymmetric key parser 'x509' registered [ 2.952354] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.954825] io scheduler mq-deadline registered [ 2.956291] io scheduler kyber registered [ 2.957360] io scheduler bfq registered [ 2.958519] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.961528] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.964338] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.967287] ACPI: Power Button [PWRF] [ 2.971723] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.977117] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.985522] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.990971] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.003425] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.028412] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.054436] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.058559] Non-volatile memory driver v1.3 [ 3.060376] Linux agpgart interface v0.103 [ 3.088073] virtio_blk virtio1: [vda] 134104 512-byte logical blocks (68.7 MB/65.5 MiB) [ 3.091733] vda: detected capacity change from 0 to 68661248 [ 3.106718] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.110516] vdb: detected capacity change from 0 to 1073741824 [ 3.128681] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.131302] vdc: detected capacity change from 0 to 2621440000 [ 3.145353] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.148233] vdd: detected capacity change from 0 to 2621440000 [ 3.166470] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.169469] vde: detected capacity change from 0 to 4294967296 [ 3.184947] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.187620] vdf: detected capacity change from 0 to 4294967296 [ 3.197491] libphy: Fixed MDIO Bus: probed [ 3.204531] usbcore: registered new interface driver usbserial_generic [ 3.207614] usbserial: USB Serial support registered for generic [ 3.210348] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.214869] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.217083] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.219773] mousedev: PS/2 mouse device common for all mice [ 3.222887] rtc_cmos 00:05: RTC can wake from S4 [ 3.225605] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.227178] rtc_cmos 00:05: registered as rtc0 [ 3.232255] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.232815] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.238610] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.238955] intel_pstate: CPU model not supported [ 3.245318] hid: raw HID events driver (C) Jiri Kosina [ 3.247057] usbcore: registered new interface driver usbhid [ 3.248608] usbhid: USB HID core driver [ 3.249895] drop_monitor: Initializing network drop monitor service [ 3.251738] Initializing XFRM netlink socket [ 3.253377] NET: Registered protocol family 10 [ 3.256275] Segment Routing with IPv6 [ 3.257446] NET: Registered protocol family 17 [ 3.259152] mpls_gso: MPLS GSO support [ 3.264709] RAS: Correctable Errors collector initialized. [ 3.266518] AVX version of gcm_enc/dec engaged. [ 3.268397] AES CTR mode by8 optimization enabled [ 3.333809] sched_clock: Marking stable (3333788310, 0)->(4242590599, -908802289) [ 3.337410] registered taskstats version 1 [ 3.339792] Loading compiled-in X.509 certificates [ 3.342309] zswap: loaded using pool lzo/zbud [ 3.362872] Key type big_key registered [ 3.372960] Key type encrypted registered [ 3.374419] ima: No TPM chip found, activating TPM-bypass! [ 3.376261] ima: Allocated hash algorithm: sha1 [ 3.378202] ima: No architecture policies found [ 3.379566] evm: Initialising EVM extended attributes: [ 3.380517] evm: security.selinux [ 3.381283] evm: security.ima [ 3.382036] evm: security.capability [ 3.383321] evm: HMAC attrs: 0x1 [ 3.385165] rtc_cmos 00:05: setting system clock to 2026-01-16 09:04:07 UTC (1768554247) [ 3.391242] debug: unmapping init [mem 0xffffffff8f603000-0xffffffff8f7fffff] [ 3.394254] debug: unmapping init [mem 0xffffffff8e382000-0xffffffff8e658fff] [ 3.402128] Write protecting the kernel read-only data: 28672k [ 3.405026] debug: unmapping init [mem 0xffffffff8ca03000-0xffffffff8cbfffff] [ 3.407202] debug: unmapping init [mem 0xffffffff8d314000-0xffffffff8d3fffff] [ 3.441692] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.450461] systemd[1]: Detected virtualization kvm. [ 3.452574] systemd[1]: Detected architecture x86-64. [ 3.454595] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.475742] systemd[1]: No hostname configured. [ 3.477574] systemd[1]: Set hostname to . [ 3.479743] random: systemd: uninitialized urandom read (16 bytes read) [ 3.482370] systemd[1]: Initializing machine ID from random generator. [ 3.520462] random: ln: uninitialized urandom read (6 bytes read) [ 3.593325] random: systemd: uninitialized urandom read (16 bytes read) [ 3.595356] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 3.598611] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.601445] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ OK ] Reached target Swap. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Initrd Root Device. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket. Starting Journal Service... [ OK ] Reached target Sockets. Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Apply Kernel Variables... [ OK ] Reached target Slices. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.103366] device-mapper: uevent: version 1.0.3 [ 4.105656] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.681477] virtio_net virtio0 ens2: renamed from eth0 [ 4.690302] random: fast init done [ 4.704959] scsi host0: ata_piix [ 4.708863] scsi host1: ata_piix [ 4.710163] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 4.712325] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.848280] dracut-initqueue[589]: RTNETLINK answers: File exists Starting nbd nbd0... [ 9.887165] random: crng init done [ 9.888546] random: 7 urandom warning(s) missed due to ratelimiting [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 9.984543] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 10.980393] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.206038] SELinux: Disabled at runtime. [ 11.270479] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.277691] systemd[1]: Detected virtualization kvm. [ 11.279204] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 11.707845] systemd[1]: initrd-switch-root.service: Succeeded. [ 11.710227] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 11.713732] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 11.716481] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 11.718614] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 11.726967] systemd[1]: Starting Journal Service... Starting Journal Service... [ 11.739734] systemd[1]: Mounting Kernel Debug File System... Mounting Kernel Debug File System... Mounting POSIX Message Queue File System... [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-getty.slice. Starting Remount Root and Kernel File Systems... Mounting Huge Pages File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Reached target rpc_pipefs.target. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Apply Kernel Variables... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on udev Control Socket. Starting Create list of required st…ce nodes for the current kernel... Activating swap /dev/disk/by-label/SWAP... Starting udev Coldplug all Devices... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File Syst[ 11.955678] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS em. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 12.166641] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.409738] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.436480] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.579895] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 12.589878] EDAC sbridge: Ver: 1.1.2 [ 14.139025] Key type dns_resolver registered [ 14.434593] NFS: Registering the id_resolver key type [ 14.436453] Key type id_resolver registered [ 14.438101] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg301-server login: [ 95.374492] libcfs: loading out-of-tree module taints kernel. [ 95.448390] Key type ._llcrypt registered [ 95.453558] Key type .llcrypt registered [ 95.588797] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_hostid [ 116.683464] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 118.697962] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 118.741778] alg: No test for adler32 (adler32-zlib) [ 120.442830] Lustre: Lustre: Build Version: 2.17.50_30_gc0571ea [ 121.276420] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [ 123.080330] Key type lgssc registered [ 124.795815] Lustre: Echo OBD driver; http://www.lustre.org/ [ 144.968410] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 194.965080] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 204.644537] hrtimer: interrupt took 1620917 ns [ 208.431521] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 208.475582] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 209.712185] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 209.750970] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 209.868281] Lustre: lustre-MDT0000: new disk, initializing [ 209.936296] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 209.954846] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 214.629939] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 228.428469] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 228.528481] Lustre: 6511:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 228.562320] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 228.565875] Lustre: Skipped 1 previous similar message [ 228.664270] Lustre: lustre-MDT0001: new disk, initializing [ 228.742758] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 228.771352] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 228.787292] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 233.455209] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 238.470744] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 250.276069] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 250.728885] Lustre: lustre-OST0000: new disk, initializing [ 250.739732] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 250.844712] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 251.351969] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 251.365283] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 251.462412] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 258.164506] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 274.701244] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 274.890092] Lustre: lustre-OST0001: new disk, initializing [ 274.896916] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 275.040739] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 282.207487] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 284.187804] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 284.197822] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 284.273775] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 296.081434] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 306.373105] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 318.936488] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing check_logdir /tmp/testlogs/ [ 324.545399] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing yml_node [ 328.894837] Lustre: DEBUG MARKER: Client: 2.17.50.30 [ 332.213248] Lustre: DEBUG MARKER: MDS: 2.17.50.30 [ 335.135097] Lustre: DEBUG MARKER: OSS: 2.17.50.30 [ 337.085784] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Jan 16 04:09:39 EST 2026 [ 357.400842] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 359.290424] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 377.313769] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 377.323426] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 377.354940] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 381.411061] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 381.414454] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 381.435726] Lustre: Skipped 2 previous similar messages [ 382.866182] LustreError: 12216:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 383.368888] Lustre: server umount lustre-MDT0000 complete [ 388.705842] LustreError: 9473:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768554632 with bad export cookie 17268264489298248291 [ 388.717157] LustreError: 9473:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 388.722745] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 391.649577] LustreError: 6517:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 391.650864] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 391.660454] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 391.660462] Lustre: Skipped 3 previous similar messages [ 391.679263] LustreError: 6517:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 391.709167] Lustre: Skipped 1 previous similar message [ 393.697502] LustreError: 6519:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 393.698646] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 393.741552] LustreError: 6519:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 393.767141] Lustre: Skipped 2 previous similar messages [ 394.977025] LustreError: 12418:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 394.987933] LustreError: 12418:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 395.727372] Lustre: server umount lustre-MDT0001 complete [ 400.828326] LustreError: 12620:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 400.836249] LustreError: 12620:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 401.042000] Lustre: server umount lustre-OST0000 complete [ 405.529706] LustreError: 12820:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 405.536930] LustreError: 12820:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 405.638916] Lustre: server umount lustre-OST0001 complete [ 414.037701] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_hostid [ 423.548508] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 468.169874] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 468.594665] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 468.666644] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 468.850101] Lustre: lustre-MDT0000: new disk, initializing [ 469.039745] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 469.091595] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 474.753024] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 486.886113] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 487.041855] Lustre: 14995:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 487.129614] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 487.134990] Lustre: Skipped 1 previous similar message [ 487.236961] Lustre: lustre-MDT0001: new disk, initializing [ 487.358629] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 487.418552] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 487.444941] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 492.913823] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 502.207604] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 506.471941] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 517.194287] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 517.708297] Lustre: lustre-OST0000: new disk, initializing [ 517.712743] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 517.852736] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 519.628109] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 519.646869] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 519.808940] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 525.390672] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 534.490569] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 539.837857] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 540.224885] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 545.040707] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 545.307708] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 550.372208] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 550.391346] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 550.439718] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 553.727199] LustreError: 17927:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 553.740789] LustreError: 17927:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 553.913085] Lustre: server umount lustre-OST0000 complete [ 559.094027] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 559.106291] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 559.129405] Lustre: Skipped 1 previous similar message [ 559.145263] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 559.149390] Lustre: Skipped 1 previous similar message [ 565.259428] LustreError: 18129:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 565.272639] LustreError: 18129:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 565.528793] Lustre: server umount lustre-MDT0000 complete [ 565.729228] LustreError: 15852:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 565.765102] LustreError: 15852:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 569.735626] LustreError: 14986:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768554813 with bad export cookie 17268264489298249817 [ 569.761939] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 570.419975] Lustre: server umount lustre-MDT0001 complete [ 575.879513] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 04:13:37 (1768554817) [ 578.629064] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 580.641442] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 04:13:42 (1768554822) [ 590.182032] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 590.739788] LustreError: 19001:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 590.876384] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 595.941424] LustreError: 19001:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 595.969791] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 605.154175] LustreError: 19000:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 605.197804] LustreError: 19000:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 605.936381] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 611.306956] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 618.864213] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 624.444728] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 633.654451] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 641.237704] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 648.027035] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 660.962991] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 660.982569] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 661.007158] Lustre: Skipped 1 previous similar message [ 661.022075] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 661.024895] Lustre: Skipped 2 previous similar messages [ 666.378134] LustreError: 21261:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 666.381187] LustreError: 21261:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 666.548885] Lustre: server umount lustre-OST0000 complete [ 678.156024] Lustre: server umount lustre-MDT0000 complete [ 682.467173] LustreError: 19000:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 682.491724] LustreError: 19000:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 683.016773] LustreError: 18981:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768554927 with bad export cookie 17268264489298250706 [ 683.023721] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 683.035423] LustreError: 18981:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 683.971570] Lustre: server umount lustre-MDT0001 complete [ 698.376511] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [ 702.568586] Key type lgssc unregistered [ 702.962664] LNet: 22200:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 702.975567] LNetError: 22200:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 702.995097] LNet: Removed LNI 192.168.203.101@tcp [ 703.948181] Key type .llcrypt unregistered [ 703.950072] Key type ._llcrypt unregistered [ 728.493097] Key type ._llcrypt registered [ 728.496181] Key type .llcrypt registered [ 728.621426] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 730.075843] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 730.236874] alg: No test for adler32 (adler32-zlib) [ 731.574922] Lustre: Lustre: Build Version: 2.17.50_30_gc0571ea [ 731.952549] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [ 733.608167] Key type lgssc registered [ 734.685807] Lustre: Echo OBD driver; http://www.lustre.org/ [ 745.891484] loop: module loaded [ 746.005776] loop0: detected capacity change from 0 to 204800000 [ 746.327096] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 746.333834] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 746.809780] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 754.304957] loop0: detected capacity change from 0 to 204800000 [ 754.542351] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 754.549875] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 754.557238] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 760.561805] loop0: detected capacity change from 0 to 204800000 [ 767.387823] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 767.440802] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 768.930663] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 769.000979] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 769.094587] Lustre: lustre-MDT0000: new disk, initializing [ 769.246580] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 769.268286] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 774.544484] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 785.217327] loop1: detected capacity change from 0 to 204800000 [ 793.043943] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 793.516175] Lustre: lustre-OST07c6: new disk, initializing [ 793.528054] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 793.552370] Lustre: Skipped 1 previous similar message [ 793.686532] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 802.153199] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 803.462218] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 803.483329] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 803.537145] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 809.365681] Lustre: 24359:0:(osd_internal.h:1459:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 809.368833] Lustre: 24359:0:(osd_handler.c:2096:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 809.373298] Lustre: 24359:0:(osd_handler.c:2103:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 809.377820] Lustre: 24359:0:(osd_handler.c:2113:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 809.382478] Lustre: 24359:0:(osd_handler.c:2120:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 809.387485] Lustre: 24359:0:(osd_handler.c:2127:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 823.001453] Lustre: 24358:0:(osd_internal.h:1459:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 823.032528] Lustre: 24358:0:(osd_handler.c:2096:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 823.045754] Lustre: 24358:0:(osd_handler.c:2103:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 823.055791] Lustre: 24358:0:(osd_handler.c:2113:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/4 [ 823.064716] Lustre: 24358:0:(osd_handler.c:2120:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 823.079993] Lustre: 24358:0:(osd_handler.c:2127:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 828.386055] Lustre: 25364:0:(osd_internal.h:1459:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 258 < left 278, rollback = 2 [ 828.406921] Lustre: 25364:0:(osd_internal.h:1459:osd_trans_exec_op()) Skipped 1 previous similar message [ 828.421280] Lustre: 25364:0:(osd_handler.c:2096:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 828.435016] Lustre: 25364:0:(osd_handler.c:2096:osd_trans_dump_creds()) Skipped 1 previous similar message [ 828.444642] Lustre: 25364:0:(osd_handler.c:2103:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 828.455968] Lustre: 25364:0:(osd_handler.c:2103:osd_trans_dump_creds()) Skipped 1 previous similar message [ 828.471115] Lustre: 25364:0:(osd_handler.c:2113:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/0 [ 828.481763] Lustre: 25364:0:(osd_handler.c:2113:osd_trans_dump_creds()) Skipped 1 previous similar message [ 828.500984] Lustre: 25364:0:(osd_handler.c:2120:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 828.510713] Lustre: 25364:0:(osd_handler.c:2120:osd_trans_dump_creds()) Skipped 1 previous similar message [ 828.524981] Lustre: 25364:0:(osd_handler.c:2127:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 828.537018] Lustre: 25364:0:(osd_handler.c:2127:osd_trans_dump_creds()) Skipped 1 previous similar message [ 833.567978] Lustre: 25364:0:(osd_internal.h:1459:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 833.586360] Lustre: 25364:0:(osd_internal.h:1459:osd_trans_exec_op()) Skipped 1 previous similar message [ 833.596578] Lustre: 25364:0:(osd_handler.c:2096:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 833.616658] Lustre: 25364:0:(osd_handler.c:2096:osd_trans_dump_creds()) Skipped 1 previous similar message [ 833.639766] Lustre: 25364:0:(osd_handler.c:2103:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 833.652264] Lustre: 25364:0:(osd_handler.c:2103:osd_trans_dump_creds()) Skipped 1 previous similar message [ 833.661041] Lustre: 25364:0:(osd_handler.c:2113:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/4 [ 833.682675] Lustre: 25364:0:(osd_handler.c:2113:osd_trans_dump_creds()) Skipped 1 previous similar message [ 833.699214] Lustre: 25364:0:(osd_handler.c:2120:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 833.711112] Lustre: 25364:0:(osd_handler.c:2120:osd_trans_dump_creds()) Skipped 1 previous similar message [ 833.722208] Lustre: 25364:0:(osd_handler.c:2127:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 833.738981] Lustre: 25364:0:(osd_handler.c:2127:osd_trans_dump_creds()) Skipped 1 previous similar message [ 839.078159] Lustre: 24359:0:(osd_internal.h:1459:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 258 < left 278, rollback = 2 [ 839.102289] Lustre: 24359:0:(osd_internal.h:1459:osd_trans_exec_op()) Skipped 1 previous similar message [ 839.118917] Lustre: 24359:0:(osd_handler.c:2096:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 839.135354] Lustre: 24359:0:(osd_handler.c:2096:osd_trans_dump_creds()) Skipped 1 previous similar message [ 839.179639] Lustre: 24359:0:(osd_handler.c:2103:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 839.205292] Lustre: 24359:0:(osd_handler.c:2103:osd_trans_dump_creds()) Skipped 1 previous similar message [ 839.231645] Lustre: 24359:0:(osd_handler.c:2113:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/0 [ 839.261680] Lustre: 24359:0:(osd_handler.c:2113:osd_trans_dump_creds()) Skipped 1 previous similar message [ 839.289253] Lustre: 24359:0:(osd_handler.c:2120:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 839.301225] Lustre: 24359:0:(osd_handler.c:2120:osd_trans_dump_creds()) Skipped 1 previous similar message [ 839.320057] Lustre: 24359:0:(osd_handler.c:2127:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 839.332306] Lustre: 24359:0:(osd_handler.c:2127:osd_trans_dump_creds()) Skipped 1 previous similar message [ 848.197705] Lustre: 24359:0:(osd_internal.h:1459:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 848.220288] Lustre: 24359:0:(osd_internal.h:1459:osd_trans_exec_op()) Skipped 3 previous similar messages [ 848.238834] Lustre: 24359:0:(osd_handler.c:2096:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 848.244828] Lustre: 24359:0:(osd_handler.c:2096:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 848.257668] Lustre: 24359:0:(osd_handler.c:2103:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 848.272286] Lustre: 24359:0:(osd_handler.c:2103:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 848.291224] Lustre: 24359:0:(osd_handler.c:2113:osd_trans_dump_creds()) write: 2/23/0, punch: 0/0/0, quota 4/150/0 [ 848.309171] Lustre: 24359:0:(osd_handler.c:2113:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 848.315846] Lustre: 24359:0:(osd_handler.c:2120:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 848.321979] Lustre: 24359:0:(osd_handler.c:2120:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 848.326483] Lustre: 24359:0:(osd_handler.c:2127:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 848.345841] Lustre: 24359:0:(osd_handler.c:2127:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 905.701249] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 905.728044] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 909.409696] LustreError: 26544:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 909.472687] Lustre: server umount lustre-OST07c6 complete [ 920.804135] LustreError: 27045:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 920.809947] LustreError: 27045:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 920.985919] Lustre: server umount lustre-MDT0000 complete [ 943.543083] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [ 947.870862] Key type lgssc unregistered [ 948.169715] LNet: 27880:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 948.175214] LNetError: 27880:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 949.227555] LNet: Removed LNI 192.168.203.101@tcp [ 950.294165] Key type .llcrypt unregistered [ 950.296181] Key type ._llcrypt unregistered [ 971.905624] Key type ._llcrypt registered [ 971.907749] Key type .llcrypt registered [ 972.097092] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 973.658046] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 973.828086] alg: No test for adler32 (adler32-zlib) [ 974.977421] Lustre: Lustre: Build Version: 2.17.50_30_gc0571ea [ 975.264930] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [ 976.960289] Key type lgssc registered [ 978.298988] Lustre: Echo OBD driver; http://www.lustre.org/ [ 986.508052] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 986.543367] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 988.164219] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 992.573958] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 998.372019] LustreError: 29132:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 998.396676] LustreError: 29132:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1002.201177] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1002.692270] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1007.905602] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 1014.064498] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1018.083704] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1025.802202] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1026.114359] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1032.463352] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 1033.415480] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 1040.237539] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1063.396510] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1063.410541] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1063.425241] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1064.419300] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1064.432235] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1064.452319] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1067.201842] LustreError: 31579:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1067.309436] Lustre: server umount lustre-OST0000 complete [ 1074.145187] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1074.159438] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1074.174461] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1074.191063] Lustre: Skipped 1 previous similar message [ 1077.406636] LustreError: 31781:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1077.414768] LustreError: 31781:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1077.619356] Lustre: server umount lustre-MDT0000 complete [ 1079.788815] LustreError: 29845:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1079.814361] LustreError: 29845:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1082.229283] LustreError: 29113:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768555326 with bad export cookie 13229914167639247391 [ 1082.243678] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1082.247409] LustreError: 29113:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1082.359282] LustreError: 31984:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1082.365686] LustreError: 31984:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1082.566989] Lustre: server umount lustre-MDT0001 complete [ 1095.521800] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [ 1099.137570] Key type lgssc unregistered [ 1099.516695] LNet: 32517:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1099.528183] LNetError: 32517:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1099.558143] LNet: Removed LNI 192.168.203.101@tcp [ 1100.356880] Key type .llcrypt unregistered [ 1100.361983] Key type ._llcrypt unregistered [ 1118.669833] Key type ._llcrypt registered [ 1118.680384] Key type .llcrypt registered [ 1118.788651] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 1119.928800] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1120.008333] alg: No test for adler32 (adler32-zlib) [ 1121.579923] Lustre: Lustre: Build Version: 2.17.50_30_gc0571ea [ 1122.070735] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [ 1123.945502] Key type lgssc registered [ 1125.420595] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1134.564908] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1134.626603] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1136.446435] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1142.065967] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 1146.851436] LustreError: 33767:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1146.879831] LustreError: 33767:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1151.978943] LustreError: 33767:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1152.179801] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1152.848465] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1158.416138] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 1165.382519] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1169.414121] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1176.752768] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1177.108552] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1181.266947] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1184.889836] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 1191.562916] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1203.682643] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1203.698083] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1203.734772] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1206.759925] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1206.776790] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1206.778391] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1208.130661] LustreError: 36009:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1208.271369] Lustre: server umount lustre-OST0000 complete [ 1214.444544] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1214.446880] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1214.452267] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1214.454357] Lustre: Skipped 1 previous similar message [ 1216.995158] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1216.995473] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1222.115751] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1222.139906] Lustre: Skipped 2 previous similar messages [ 1227.232225] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1227.436054] LustreError: 36211:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1227.451820] LustreError: 36211:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1227.606335] Lustre: server umount lustre-MDT0000 complete [ 1231.339885] LustreError: 34478:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1231.369969] LustreError: 34478:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1233.222595] LustreError: 36010:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768555477 with bad export cookie 2435130707804461968 [ 1233.226613] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1233.245173] LustreError: 36010:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1233.461239] LustreError: 36414:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1233.466362] LustreError: 36414:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1233.791574] Lustre: server umount lustre-MDT0001 complete [ 1249.156644] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [ 1252.905861] Key type lgssc unregistered [ 1253.160308] LNet: 36949:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1253.175904] LNetError: 36949:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1253.206973] LNet: Removed LNI 192.168.203.101@tcp [ 1254.674890] Key type .llcrypt unregistered [ 1254.677523] Key type ._llcrypt unregistered [ 1286.412609] Key type ._llcrypt registered [ 1286.416438] Key type .llcrypt registered [ 1286.587761] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 1288.113834] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1288.333167] alg: No test for adler32 (adler32-zlib) [ 1289.585801] Lustre: Lustre: Build Version: 2.17.50_30_gc0571ea [ 1289.906974] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [ 1291.754277] Key type lgssc registered [ 1292.942238] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1303.324034] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1303.363248] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1305.204615] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1312.183790] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 1315.819972] LustreError: 38349:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1315.857989] LustreError: 38349:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1320.929780] LustreError: 38349:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1323.762753] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1324.410708] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1329.458956] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 1336.565951] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1340.803488] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1348.182306] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1348.568517] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1350.642693] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1355.945388] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 1363.400492] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1371.412540] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.1@tcp (stopping) [ 1375.713755] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1375.724833] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1375.746544] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1376.240654] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1377.534168] LustreError: 40595:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1377.910115] Lustre: server umount lustre-MDT0000 complete [ 1381.357688] LustreError: 38346:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1381.370604] LustreError: 38346:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1382.860853] LustreError: 38331:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768555626 with bad export cookie 15609686124971778188 [ 1382.861529] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1382.870990] LustreError: 38331:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1386.465681] LustreError: 38346:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1386.475245] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1386.480536] LustreError: 38346:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1386.521515] Lustre: Skipped 1 previous similar message [ 1386.527216] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1386.531581] Lustre: Skipped 4 previous similar messages [ 1391.836891] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.1@tcp (stopping) [ 1391.836901] LustreError: 38346:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.1@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1391.846246] Lustre: Skipped 2 previous similar messages [ 1391.877703] LustreError: 38346:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1396.965685] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.1@tcp (stopping) [ 1396.973653] Lustre: Skipped 2 previous similar messages [ 1397.216172] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1397.367528] LustreError: 40797:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1397.382162] LustreError: 40797:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1397.568528] Lustre: server umount lustre-MDT0001 complete [ 1413.249239] LustreError: 41020:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1413.263126] LustreError: 41020:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1413.397961] Lustre: server umount lustre-OST0000 complete [ 1431.622530] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [ 1435.543544] Key type lgssc unregistered [ 1435.895988] LNet: 41858:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1435.907743] LNetError: 41858:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1435.923237] LNet: Removed LNI 192.168.203.101@tcp [ 1436.787698] Key type .llcrypt unregistered [ 1436.789837] Key type ._llcrypt unregistered [ 1462.199668] Key type ._llcrypt registered [ 1462.202806] Key type .llcrypt registered [ 1462.312729] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 1463.577724] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1463.624090] alg: No test for adler32 (adler32-zlib) [ 1464.851628] Lustre: Lustre: Build Version: 2.17.50_30_gc0571ea [ 1465.246837] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [ 1466.920163] Key type lgssc registered [ 1468.409619] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1477.999930] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1478.036725] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1479.635493] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1484.859545] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 1489.894215] LustreError: 43255:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1489.923050] LustreError: 43255:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1495.012125] LustreError: 43255:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1495.164342] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1495.654841] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1501.069776] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 1508.508546] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1513.369556] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1521.485155] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1522.085925] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1525.207214] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1527.609730] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 1533.621937] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1545.697116] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1545.716270] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1545.721655] LustreError: Skipped 1 previous similar message [ 1545.725923] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1545.778663] Lustre: Skipped 1 previous similar message [ 1549.238678] LustreError: 45500:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1549.420827] Lustre: server umount lustre-OST0000 complete [ 1570.288473] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1570.292949] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1570.306104] Lustre: Skipped 1 previous similar message [ 1570.326171] Lustre: Skipped 3 previous similar messages [ 1575.393897] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1575.398527] Lustre: Skipped 1 previous similar message [ 1580.513357] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1582.050923] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1582.233996] LustreError: 45875:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1582.247220] LustreError: 45875:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1582.698106] Lustre: server umount lustre-MDT0000 complete [ 1585.633522] LustreError: 44463:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1585.662114] LustreError: 44463:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1587.467586] LustreError: 44873:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768555831 with bad export cookie 15296248328119818299 [ 1587.474171] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1587.476024] LustreError: 44873:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1593.766690] LustreError: 46077:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1593.779302] LustreError: 46077:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1594.071617] Lustre: server umount lustre-MDT0001 complete [ 1606.979919] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [ 1610.331865] Key type lgssc unregistered [ 1610.692251] LNet: 46612:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1610.697438] LNetError: 46612:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1610.724179] LNet: Removed LNI 192.168.203.101@tcp [ 1611.844639] Key type .llcrypt unregistered [ 1611.845889] Key type ._llcrypt unregistered [ 1640.512756] Key type ._llcrypt registered [ 1640.521798] Key type .llcrypt registered [ 1640.646735] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 1642.247212] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1642.323251] alg: No test for adler32 (adler32-zlib) [ 1643.649166] Lustre: Lustre: Build Version: 2.17.50_30_gc0571ea [ 1643.983835] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [ 1645.696167] Key type lgssc registered [ 1647.352796] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1656.567145] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1656.598075] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1658.418868] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1665.107142] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 1668.588144] LustreError: 48008:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1668.617215] LustreError: 48008:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1673.710755] LustreError: 48008:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1678.344242] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1678.939243] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1684.827384] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 1691.698413] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1696.009667] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1705.696879] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1706.237067] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1715.177996] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 1721.985824] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1728.781759] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1738.851490] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1742.956892] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1755.617594] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-01-16 4h33m16s [ 1757.524791] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1760.753753] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1760.772256] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1760.812202] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1761.893351] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1761.901946] Lustre: Skipped 4 previous similar messages [ 1764.098172] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.1@tcp (stopping) [ 1766.335286] LustreError: 50594:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1766.523865] Lustre: server umount lustre-MDT0000 complete [ 1768.441596] LustreError: 50596:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1768.460740] LustreError: 50596:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1770.951432] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1771.230668] LustreError: 50596:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.1@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1777.377179] LustreError: 48004:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.1@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1777.395399] LustreError: 48004:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 1777.448532] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1777.693019] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1777.953561] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1778.023755] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1779.044672] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1783.068693] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 1787.672924] Lustre: DEBUG MARKER: Wait for df (36128) ... [ 1790.000444] Lustre: DEBUG MARKER: done [ 1795.554366] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1795.565570] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1795.580705] Lustre: Skipped 2 previous similar messages [ 1795.591157] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1795.600544] Lustre: Skipped 1 previous similar message [ 1797.602136] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1802.726850] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1802.734712] Lustre: Skipped 3 previous similar messages [ 1809.376137] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1809.417168] LustreError: 51651:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1809.419242] LustreError: 51651:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1809.649499] Lustre: server umount lustre-OST0000 complete [ 1815.016530] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1815.029181] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1815.062957] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1815.065267] Lustre: Skipped 2 previous similar messages [ 1820.619475] LustreError: 51855:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1820.624611] LustreError: 51855:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1820.838252] Lustre: server umount lustre-MDT0000 complete [ 1824.747783] LustreError: 48007:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1824.775107] LustreError: 48007:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1824.803785] LustreError: 50323:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768556068 with bad export cookie 4524723867344111213 [ 1824.807082] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1824.812850] LustreError: 50323:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1825.005497] LustreError: 52056:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1825.012237] LustreError: 52056:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1825.172916] Lustre: server umount lustre-MDT0001 complete [ 1839.266650] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [ 1843.784561] Key type lgssc unregistered [ 1844.371374] LNet: 52589:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1844.383605] LNetError: 52589:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1844.424810] LNet: Removed LNI 192.168.203.101@tcp [ 1846.152852] Key type .llcrypt unregistered [ 1846.156140] Key type ._llcrypt unregistered [ 1892.588974] Key type ._llcrypt registered [ 1892.591156] Key type .llcrypt registered [ 1892.721851] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 1894.233974] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1894.323279] alg: No test for adler32 (adler32-zlib) [ 1895.382426] Lustre: Lustre: Build Version: 2.17.50_30_gc0571ea [ 1895.573792] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [ 1897.288472] Key type lgssc registered [ 1898.891370] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1907.932832] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1907.975043] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1909.429051] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1909.491082] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1909.774849] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1915.525605] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 1925.687775] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1925.910950] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1925.949374] Lustre: 54782:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1926.194277] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1931.349621] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 1938.849968] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1943.979987] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1951.554674] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1951.853911] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1952.145911] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1958.593219] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1958.926697] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 1965.159954] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1972.117841] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-01-16 4h36m54s [ 1973.856044] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1981.973152] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1984.891605] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1984.948718] LustreError: 56013:0:(mdt_handler.c:2334:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 2005.289488] Lustre: lustre-MDT0000: Client 604d39aa-3d3c-45ca-95a4-347971c7a0e4 (at 192.168.203.1@tcp) reconnecting [ 2024.992151] LustreError: 56013:0:(mdt_handler.c:2334:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 2028.186548] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 2031.217852] Lustre: DEBUG MARKER: done [ 2040.290372] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2040.300877] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2040.322225] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2040.807609] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2040.839506] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2045.330439] LustreError: 57820:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2045.479686] Lustre: server umount lustre-OST0000 complete [ 2051.041872] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2051.048525] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2051.073544] Lustre: Skipped 1 previous similar message [ 2051.095904] Lustre: Skipped 1 previous similar message [ 2058.209648] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2058.219099] Lustre: Skipped 3 previous similar messages [ 2062.310393] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2062.316630] Lustre: Skipped 1 previous similar message [ 2064.352202] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2064.481362] LustreError: 58022:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2064.503102] LustreError: 58022:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2064.626932] Lustre: server umount lustre-MDT0000 complete [ 2068.672741] LustreError: 54776:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768556312 with bad export cookie 2503073147906144135 [ 2068.678530] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2068.690556] LustreError: 54776:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2068.882915] LustreError: 58224:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2068.888507] LustreError: 58224:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2069.257644] Lustre: server umount lustre-MDT0001 complete [ 2082.396371] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [ 2085.430565] Key type lgssc unregistered [ 2085.681225] LNet: 58757:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2085.690683] LNetError: 58757:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2085.705853] LNet: Removed LNI 192.168.203.101@tcp [ 2086.615106] Key type .llcrypt unregistered [ 2086.617386] Key type ._llcrypt unregistered [ 2137.787907] Key type ._llcrypt registered [ 2137.805252] Key type .llcrypt registered [ 2137.945343] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 2139.711798] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2139.802118] alg: No test for adler32 (adler32-zlib) [ 2140.963807] Lustre: Lustre: Build Version: 2.17.50_30_gc0571ea [ 2141.248161] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [ 2142.984834] Key type lgssc registered [ 2144.529659] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2149.432592] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2149.466056] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2150.881504] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2150.925913] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2151.243710] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2152.877424] Lustre: Failing over lustre-MDT0000 [ 2153.058020] LustreError: 60983:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2153.399697] Lustre: server umount lustre-MDT0000 complete [ 2164.090115] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 04:40:06 (1768556406) [ 2172.195238] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2172.783615] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2177.592440] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 2186.344797] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2186.498687] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2186.536327] Lustre: 61540:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2186.994705] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2192.868241] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 2199.526433] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2204.469653] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2212.520952] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2212.768909] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2213.036775] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2218.577576] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 2222.687633] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2225.260042] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2231.041722] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2238.440067] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2238.458727] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2238.490584] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2243.048316] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2243.050633] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2243.071161] Lustre: Skipped 1 previous similar message [ 2243.080709] Lustre: Skipped 1 previous similar message [ 2244.287101] LustreError: 63909:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2244.293831] LustreError: 63909:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2244.565613] Lustre: server umount lustre-MDT0000 complete [ 2249.042812] LustreError: 62257:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768556493 with bad export cookie 130490680797105119 [ 2249.056055] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2249.063256] LustreError: 62257:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2249.227992] LustreError: 64112:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2249.237675] LustreError: 64112:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2249.483777] Lustre: server umount lustre-MDT0001 complete [ 2253.867981] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2259.289663] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2269.665721] Lustre: 60463:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1768556497/real 1768556497] req@0000000089b80874 x1854465779998976/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1768556513 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2269.685564] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2269.710416] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2270.696324] Lustre: 60462:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1768556498/real 1768556498] req@0000000053fb2771 x1854465779999104/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1768556514 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2274.791442] Lustre: 60461:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1768556502/real 1768556502] req@00000000344837c1 x1854465779999232/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1768556518 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2275.253692] LustreError: 63163:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2275.271118] LustreError: 63163:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2275.349467] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2275.434913] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2278.371573] LustreError: 63163:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2279.394692] LustreError: 63163:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2279.412226] LustreError: 63163:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2279.951736] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 2280.419172] Lustre: 60462:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1768556508/real 1768556508] req@00000000bd929436 x1854465779999616/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1768556524 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2280.455225] Lustre: 60462:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2284.516821] LustreError: 64970:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2284.544511] LustreError: 64970:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2285.540085] Lustre: 60462:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1768556513/real 1768556513] req@00000000a850660d x1854465779999744/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1768556529 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2289.646757] LustreError: 63162:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2289.690735] LustreError: 63162:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2290.683873] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2291.145832] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2296.317820] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2297.582665] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 2298.408246] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2304.213177] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2308.070974] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2316.770376] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2316.777863] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2316.798948] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2316.807590] Lustre: Skipped 1 previous similar message [ 2318.824390] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2321.647201] LustreError: 66371:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2321.654293] LustreError: 66371:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2321.821920] Lustre: server umount lustre-MDT0000 complete [ 2323.940612] LustreError: 65690:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2323.961296] LustreError: 65690:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2326.137230] LustreError: 61533:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768556570 with bad export cookie 130490680797107394 [ 2326.144953] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2326.151395] LustreError: 61533:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2326.382586] LustreError: 66572:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2326.400236] LustreError: 66572:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2326.646753] Lustre: server umount lustre-MDT0001 complete [ 2334.754889] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2345.376148] Lustre: 60462:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1768556573/real 1768556573] req@00000000a00ac267 x1854465780028288/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1768556589 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2345.414208] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2345.444125] Lustre: Skipped 2 previous similar messages [ 2346.979223] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2351.589438] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x1cf989533691a1e [ 2351.618526] Lustre: MGC192.168.203.101@tcp: Connection restored to 0@lo (at 0@lo) [ 2352.288031] LustreError: 65690:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2352.378946] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2352.435239] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2356.822343] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 2367.884746] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2372.541854] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 2373.656125] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2379.479737] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2383.884978] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2394.084659] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2394.089381] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2394.109508] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2394.116541] Lustre: Skipped 2 previous similar messages [ 2397.844987] LustreError: 68825:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2397.862930] LustreError: 68825:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2398.207924] Lustre: server umount lustre-MDT0000 complete [ 2400.227694] LustreError: 67428:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2400.269340] LustreError: 67428:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 2402.600275] LustreError: 62257:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768556646 with bad export cookie 130490680797108766 [ 2402.608430] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2402.934713] Lustre: server umount lustre-MDT0001 complete [ 2410.549875] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2415.273134] Lustre: DEBUG MARKER: files compared the same [ 2427.050371] LustreError: 69623:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2427.062729] LustreError: 69623:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 2427.373277] Lustre: server umount lustre-OST0000 complete [ 2446.593959] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [ 2449.820475] Key type lgssc unregistered [ 2450.086703] LNet: 70464:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2450.095591] LNetError: 70464:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2450.124241] LNet: Removed LNI 192.168.203.101@tcp [ 2451.073550] Key type .llcrypt unregistered [ 2451.076814] Key type ._llcrypt unregistered [ 2474.709789] Key type ._llcrypt registered [ 2474.712972] Key type .llcrypt registered [ 2474.853423] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 2476.335267] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2476.464050] alg: No test for adler32 (adler32-zlib) [ 2477.651883] Lustre: Lustre: Build Version: 2.17.50_30_gc0571ea [ 2477.926494] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [ 2479.657056] Key type lgssc registered [ 2481.278621] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2490.967877] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2491.041519] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2492.786388] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2498.158693] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug +malloc all [ 2502.120656] LustreError: 71861:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2502.134072] LustreError: 71861:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2507.240904] LustreError: 71861:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2508.855256] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2509.472379] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2516.094484] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug +malloc all [ 2522.855618] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2528.117592] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2535.787369] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2535.977222] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2537.007553] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2543.489862] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug +malloc all [ 2550.467495] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2564.578794] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2564.581161] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2564.585156] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2569.708269] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2569.725085] Lustre: Skipped 1 previous similar message [ 2570.511155] LustreError: 74126:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2570.740058] Lustre: server umount lustre-OST0000 complete [ 2575.840844] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2575.853247] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2575.868196] Lustre: Skipped 1 previous similar message [ 2575.889422] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2575.901451] Lustre: Skipped 1 previous similar message [ 2579.939874] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2579.941927] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2579.967185] Lustre: Skipped 1 previous similar message [ 2581.078196] LustreError: 74327:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2581.084392] LustreError: 74327:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2581.197234] Lustre: server umount lustre-MDT0000 complete [ 2584.937664] LustreError: 71840:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768556829 with bad export cookie 6936966850989106829 [ 2584.942344] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2584.949229] LustreError: 71840:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2585.128693] LustreError: 74528:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2585.136818] LustreError: 74528:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2585.580634] Lustre: server umount lustre-MDT0001 complete [ 2597.942147] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [ 2601.875231] Key type lgssc unregistered [ 2602.160515] LNet: 75061:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2602.172165] LNetError: 75061:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2602.193827] LNet: Removed LNI 192.168.203.101@tcp [ 2603.138496] Key type .llcrypt unregistered [ 2603.142584] Key type ._llcrypt unregistered [ 2630.321042] Key type ._llcrypt registered [ 2630.323566] Key type .llcrypt registered [ 2630.485941] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 2631.964905] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2632.094175] alg: No test for adler32 (adler32-zlib) [ 2633.355906] Lustre: Lustre: Build Version: 2.17.50_30_gc0571ea [ 2633.700659] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [ 2635.428384] Key type lgssc registered [ 2636.792192] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2645.597948] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2645.615970] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2661.152761] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2666.272334] LustreError: 76433:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.101@tcp: failed processing log, type 4: rc = -110 [ 2693.088332] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2699.703913] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 2707.293936] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2716.698785] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2716.896381] Lustre: *** cfs_fail_loc=706, val=0*** [ 2717.498453] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2717.608356] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2722.634628] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 2722.785800] LustreError: 76459:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2722.806095] LustreError: 76459:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2727.912202] LustreError: 76459:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2727.930362] LustreError: 76459:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2732.491721] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2732.953557] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2737.863367] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 2744.055879] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2748.969614] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2753.525872] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2753.527141] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2753.528031] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2753.567763] Lustre: Skipped 1 previous similar message [ 2757.496416] LustreError: 78753:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 2757.751469] Lustre: server umount lustre-OST0000 complete [ 2763.744935] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2763.749091] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2763.751939] LustreError: Skipped 1 previous similar message [ 2763.753398] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2763.753406] Lustre: Skipped 1 previous similar message [ 2763.796413] Lustre: Skipped 1 previous similar message [ 2768.115869] LustreError: 78957:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 2768.131035] LustreError: 78957:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2768.254555] Lustre: server umount lustre-MDT0000 complete [ 2768.870444] LustreError: 77408:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2768.898984] LustreError: 77408:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2772.305719] LustreError: 76441:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768557016 with bad export cookie 6910652254742589121 [ 2772.310996] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2772.336849] LustreError: 76441:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2772.466629] LustreError: 79160:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2772.472149] LustreError: 79160:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2772.614515] Lustre: server umount lustre-MDT0001 complete [ 2785.566559] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [ 2789.037317] Key type lgssc unregistered [ 2789.274692] LNet: 79693:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2789.281491] LNetError: 79693:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2789.299345] LNet: Removed LNI 192.168.203.101@tcp [ 2790.131659] Key type .llcrypt unregistered [ 2790.133284] Key type ._llcrypt unregistered [ 2815.698900] Key type ._llcrypt registered [ 2815.700756] Key type .llcrypt registered [ 2815.836851] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 2817.249421] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2817.357988] alg: No test for adler32 (adler32-zlib) [ 2818.421384] Lustre: Lustre: Build Version: 2.17.50_30_gc0571ea [ 2818.656290] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [ 2820.360567] Key type lgssc registered [ 2821.383797] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2829.743597] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2829.778154] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2836.091481] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 2845.022765] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2845.572776] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2850.347400] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 2850.784989] LustreError: 81778:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2850.819313] LustreError: 81778:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2855.918206] LustreError: 81782:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2855.948633] LustreError: 81782:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2860.120827] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2860.635704] LustreError: 81782:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2860.660974] LustreError: 81782:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2860.688265] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2866.160229] LustreError: 81782:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2866.186944] LustreError: 81782:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2868.478457] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 2871.267195] LustreError: 81782:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2871.289742] LustreError: 81782:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2877.986054] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2877.994448] Lustre: MGS: Client 59a96467-d5e8-4719-83b7-fa080d448690 (at 0@lo) reconnecting [ 2878.243173] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2878.328055] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2883.621331] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 2900.450636] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2900.471088] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2900.490619] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2904.041717] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2904.051958] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2904.058974] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2904.113582] Lustre: Skipped 1 previous similar message [ 2905.674316] LustreError: 83757:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 2905.901531] Lustre: server umount lustre-OST0000 complete [ 2914.273598] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2914.295942] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2914.308634] Lustre: Skipped 1 previous similar message [ 2916.236168] LustreError: 83960:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2916.245944] LustreError: 83960:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2916.311584] Lustre: server umount lustre-MDT0000 complete [ 2919.394645] LustreError: 81783:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2919.416140] LustreError: 81783:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 2920.849661] LustreError: 84162:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2920.859323] LustreError: 84162:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2921.145963] Lustre: server umount lustre-MDT0001 complete [ 2925.537665] LustreError: 84365:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2925.542622] LustreError: 84365:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2925.607859] Lustre: server umount MGS complete [ 2936.799537] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 04:52:59 (1768557179) [ 2972.993597] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_hostid [ 2986.566896] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 3035.534437] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3041.429414] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 3054.038489] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3054.340217] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 3054.388191] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3054.487584] Lustre: lustre-MDT0001: new disk, initializing [ 3054.651067] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3054.700192] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3054.711733] Lustre: 88873:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 3054.722183] LustreError: 88873:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 3060.511365] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 3073.244592] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3073.696702] Lustre: lustre-OST0000: new disk, initializing [ 3073.701784] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3073.796124] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3080.995514] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 3081.832914] Lustre: 89945:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@0000000045fe74fa x1854466489886720/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1768557341 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 3090.237389] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3098.350200] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3098.360075] Lustre: MGS: Client 1fb892fd-10cf-40d6-843f-3d54e7c9b123 (at 0@lo) reconnecting [ 3098.390848] Lustre: 88158:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3098.409897] LustreError: 88158:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x0 : rc = -11 [ 3098.440580] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3098.554088] Lustre: lustre-MDT0000: new disk, initializing [ 3098.722839] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3098.746353] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3098.761772] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3099.812474] Lustre: 90580:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@000000004d78e4b1 x1854466489893376/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1768557359 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 3103.667894] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 3104.901169] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3104.920983] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3105.110250] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 3134.946469] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3134.971888] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3135.002898] Lustre: Skipped 1 previous similar message [ 3135.019918] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3137.098078] LustreError: 91099:0:(obd_class.h:479:obd_check_dev()) Device 11 not setup [ 3137.101579] LustreError: 91099:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 3137.373283] Lustre: server umount lustre-OST0000 complete [ 3145.697506] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3145.699869] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3145.717112] Lustre: Skipped 2 previous similar messages [ 3145.728234] Lustre: Skipped 2 previous similar messages [ 3148.352605] LustreError: 91302:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 3148.359852] LustreError: 91302:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3148.466765] Lustre: server umount lustre-MDT0000 complete [ 3150.818088] LustreError: 88844:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3150.841316] LustreError: 88844:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3159.503894] Lustre: server umount lustre-MDT0001 complete [ 3164.329118] Lustre: server umount MGS complete [ 3178.655561] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 04:57:00 (1768557420) [ 3201.852699] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [ 3204.461429] Key type lgssc unregistered [ 3204.721889] LNet: 93046:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3204.729920] LNetError: 93046:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3205.807824] LNet: Removed LNI 192.168.203.101@tcp [ 3207.066714] Key type .llcrypt unregistered [ 3207.068594] Key type ._llcrypt unregistered [ 3227.437737] Key type ._llcrypt registered [ 3227.444713] Key type .llcrypt registered [ 3227.673394] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 3229.091074] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3229.178082] alg: No test for adler32 (adler32-zlib) [ 3230.591075] Lustre: Lustre: Build Version: 2.17.50_30_gc0571ea [ 3230.843329] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [ 3232.608171] Key type lgssc registered [ 3234.160801] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3243.530662] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3243.549605] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3245.246152] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3251.005180] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 3255.784724] LustreError: 94251:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3255.813125] LustreError: 94251:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3260.382649] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3260.784912] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3264.989659] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 3272.380276] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3277.275384] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3285.667368] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3285.992649] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3290.110366] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 3292.722391] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 3299.306547] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3315.684618] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3315.686532] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3315.691712] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3315.711110] Lustre: Skipped 1 previous similar message [ 3320.267815] LustreError: 96516:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3320.445784] Lustre: server umount lustre-OST0000 complete [ 3325.930779] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3325.947809] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3325.949786] Lustre: Skipped 1 previous similar message [ 3331.045698] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3331.052903] Lustre: Skipped 1 previous similar message [ 3331.367133] LustreError: 96718:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3331.375453] LustreError: 96718:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3331.465897] Lustre: server umount lustre-MDT0000 complete [ 3335.146578] LustreError: 94252:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3335.167716] LustreError: 94252:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3335.939078] LustreError: 94232:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768557580 with bad export cookie 18226533725402970494 [ 3335.955729] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3336.117891] LustreError: 96919:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 3336.125541] LustreError: 96919:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3336.454726] Lustre: server umount lustre-MDT0001 complete [ 3350.131142] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [ 3353.495989] Key type lgssc unregistered [ 3353.805811] LNet: 97453:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3353.812116] LNetError: 97453:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3353.847272] LNet: Removed LNI 192.168.203.101@tcp [ 3354.765107] Key type .llcrypt unregistered [ 3354.766703] Key type ._llcrypt unregistered [ 3373.037796] Key type ._llcrypt registered [ 3373.040847] Key type .llcrypt registered [ 3373.265695] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 3374.903093] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3374.939135] alg: No test for adler32 (adler32-zlib) [ 3376.490389] Lustre: Lustre: Build Version: 2.17.50_30_gc0571ea [ 3376.838302] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [ 3378.568248] Key type lgssc registered [ 3379.772103] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3386.808454] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3386.813526] LustreError: 98516:0:(tgt_mount.c:2439:server_fill_super()) cfs_race id 716 sleeping [ 3386.814403] LustreError: 98512:0:(tgt_mount.c:2439:server_fill_super()) cfs_fail_race id 716 waking [ 3386.823047] LustreError: 98516:0:(tgt_mount.c:2439:server_fill_super()) cfs_fail_race id 716 awake: rc=4994 [ 3386.828506] LustreError: 98516:0:(tgt_mount.c:2451:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3386.839553] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3386.847968] LustreError: 98516:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3388.475588] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3397.284406] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3397.918886] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3403.258789] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 3409.864366] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing lsmod [ 3413.307460] LustreError: 99682:0:(tgt_mount.c:2439:server_fill_super()) cfs_race id 716 sleeping [ 3413.330429] LustreError: 99689:0:(tgt_mount.c:2439:server_fill_super()) cfs_fail_race id 716 waking [ 3413.339759] LustreError: 99682:0:(tgt_mount.c:2439:server_fill_super()) cfs_fail_race id 716 awake: rc=4975 [ 3413.353816] LustreError: 99682:0:(tgt_mount.c:2451:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3413.363032] LustreError: 99682:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3413.376890] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3413.935162] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3414.965214] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3418.598522] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3418.618376] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3418.650610] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3420.130983] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3420.143357] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3420.156383] Lustre: Skipped 1 previous similar message [ 3420.178234] Lustre: Skipped 2 previous similar messages [ 3422.259231] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3422.277140] Lustre: Skipped 2 previous similar messages [ 3423.617717] LustreError: 99886:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3423.652966] LustreError: 99886:0:(ldlm_resource.c:981:ldlm_resource_complain()) MGS: namespace resource [0x65727473756c:0x0:0x0].0x0 (0000000055d98aa0) refcount nonzero (2) after lock cleanup; forcing cleanup. [ 3423.927157] Lustre: server umount lustre-MDT0000 complete [ 3427.302056] LustreError: 98554:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3427.313984] LustreError: 98554:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 3428.018824] LustreError: 98532:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768557672 with bad export cookie 18321805950802302840 [ 3428.021557] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3428.030906] LustreError: 98532:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3428.416300] LustreError: 98985:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000af0de193 x1854467074971776/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3428.466632] LustreError: 100090:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 3428.471646] LustreError: 100090:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3428.660743] Lustre: server umount lustre-MDT0001 complete [ 3442.833967] LustreError: 100293:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 3442.843304] LustreError: 100293:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3443.099280] Lustre: server umount lustre-OST0000 complete [ 3453.676606] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3454.246977] LustreError: 100770:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3454.351753] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3459.559234] LustreError: 100770:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3459.724464] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 3464.681039] LustreError: 100770:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3469.690321] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3469.799238] LustreError: 100770:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3470.260763] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3475.233505] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 3481.872812] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3486.263376] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3494.991958] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3495.422243] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3499.526310] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:97) [ 3502.715904] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 3509.512404] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3519.945488] LustreError: 103014:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3519.951611] LustreError: 103014:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3519.978732] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3519.986380] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3520.062854] LustreError: 102364:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3520.081470] LustreError: 102364:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3520.280956] Lustre: server umount lustre-OST0000 complete [ 3525.089358] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3525.102307] Lustre: Skipped 1 previous similar message [ 3525.114530] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3530.209711] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3530.212503] Lustre: Skipped 2 previous similar messages [ 3530.780504] LustreError: 103215:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3530.792255] LustreError: 103215:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3531.045593] Lustre: server umount lustre-MDT0000 complete [ 3535.192805] LustreError: 100749:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768557779 with bad export cookie 18321805950802303771 [ 3535.207681] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3535.676243] Lustre: server umount lustre-MDT0001 complete [ 3546.672533] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [ 3550.364134] Key type lgssc unregistered [ 3550.783764] LNet: 103951:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3550.795570] LNetError: 103951:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3550.814718] LNet: Removed LNI 192.168.203.101@tcp [ 3552.066547] Key type .llcrypt unregistered [ 3552.070792] Key type ._llcrypt unregistered [ 3579.941369] Key type ._llcrypt registered [ 3579.943650] Key type .llcrypt registered [ 3580.077818] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 3581.269933] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3581.447052] alg: No test for adler32 (adler32-zlib) [ 3582.670532] Lustre: Lustre: Build Version: 2.17.50_30_gc0571ea [ 3582.924233] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [ 3584.632266] Key type lgssc registered [ 3586.007727] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3595.836986] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3595.884904] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3597.473690] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3603.075665] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 3607.013712] LustreError: 105350:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3607.029746] LustreError: 105350:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3612.040775] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3612.131579] LustreError: 105350:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3612.559235] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3617.087107] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 3625.010102] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3629.593726] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3637.424174] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3638.053804] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3642.189426] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3644.686607] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 3650.944832] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3660.232704] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3663.716716] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3673.058174] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3673.061473] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3673.078363] Lustre: Skipped 1 previous similar message [ 3675.013943] LustreError: 107701:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3675.270425] Lustre: server umount lustre-OST0000 complete [ 3683.311047] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3683.314899] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3683.323335] Lustre: Skipped 1 previous similar message [ 3683.331758] Lustre: Skipped 1 previous similar message [ 3685.637889] LustreError: 107902:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3685.650699] LustreError: 107902:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3685.820672] Lustre: server umount lustre-MDT0000 complete [ 3688.420712] LustreError: 106072:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3688.460530] LustreError: 106072:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3689.514664] LustreError: 107547:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768557933 with bad export cookie 6245068724946135895 [ 3689.529372] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3689.534612] LustreError: 107547:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3689.726811] LustreError: 108104:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 3689.741777] LustreError: 108104:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3689.852510] Lustre: server umount lustre-MDT0001 complete [ 3700.719040] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [ 3703.758275] Key type lgssc unregistered [ 3703.973369] LNet: 108636:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3703.978861] LNetError: 108636:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3703.995345] LNet: Removed LNI 192.168.203.101@tcp [ 3704.699949] Key type .llcrypt unregistered [ 3704.701872] Key type ._llcrypt unregistered [ 3726.427191] Key type ._llcrypt registered [ 3726.428680] Key type .llcrypt registered [ 3726.534407] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 3727.989688] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3728.065474] alg: No test for adler32 (adler32-zlib) [ 3729.689193] Lustre: Lustre: Build Version: 2.17.50_30_gc0571ea [ 3729.925698] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [ 3731.600211] Key type lgssc registered [ 3733.146728] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3742.426408] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3742.443295] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3743.904729] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3748.811488] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 3753.442270] LustreError: 109837:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3753.465151] LustreError: 109837:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3758.564674] LustreError: 109837:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3759.053693] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3759.626148] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3764.502534] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 3771.229455] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3775.888382] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3783.919639] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3784.170250] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3788.274311] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3791.432187] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 3797.034076] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3808.738796] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3808.747426] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3808.755225] LustreError: Skipped 1 previous similar message [ 3808.757048] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3808.792250] Lustre: Skipped 1 previous similar message [ 3812.355585] LustreError: 112080:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3812.610721] Lustre: server umount lustre-OST0000 complete [ 3818.982418] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3819.011919] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3819.024497] Lustre: Skipped 2 previous similar messages [ 3822.722481] LustreError: 112281:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3822.727633] LustreError: 112281:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3822.903917] Lustre: server umount lustre-MDT0000 complete [ 3824.097145] LustreError: 110795:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3824.114906] LustreError: 110795:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3826.713956] LustreError: 109817:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768558070 with bad export cookie 1435445690457777494 [ 3826.719439] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3826.724221] LustreError: 109817:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3826.899792] LustreError: 112483:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 3826.910551] LustreError: 112483:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3827.104611] Lustre: server umount lustre-MDT0001 complete [ 3840.443153] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [ 3843.538757] Key type lgssc unregistered [ 3843.827176] LNet: 113016:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3843.833083] LNetError: 113016:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3843.852545] LNet: Removed LNI 192.168.203.101@tcp [ 3844.953283] Key type .llcrypt unregistered [ 3844.954937] Key type ._llcrypt unregistered [ 3870.836224] Key type ._llcrypt registered [ 3870.839755] Key type .llcrypt registered [ 3870.993552] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 3871.775617] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3871.827538] alg: No test for adler32 (adler32-zlib) [ 3873.030358] Lustre: Lustre: Build Version: 2.17.50_30_gc0571ea [ 3873.259911] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [ 3874.960153] Key type lgssc registered [ 3876.081889] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3884.332836] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3884.351526] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3885.939623] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3891.637028] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 3896.311900] LustreError: 114415:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3896.353773] LustreError: 114415:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3901.414502] LustreError: 114415:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3903.709143] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3904.108166] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3908.313234] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 3914.832259] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3919.422335] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3927.602767] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3928.057510] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3931.194569] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:163 to 0x280000400:193) [ 3935.971966] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 3942.124991] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3949.738377] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3953.127989] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3953.132502] Lustre: Skipped 2 previous similar messages [ 3957.171158] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3957.175978] Lustre: Skipped 2 previous similar messages [ 3957.312783] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3957.351937] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3967.911962] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.203.1@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3967.920779] Lustre: Skipped 2 previous similar messages [ 3974.682943] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.203.1@tcp 192.168.0.[2,10]@tcp [ 3978.846617] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.203.1/24@tcp in log lustre-MDT0000 [ 3978.851082] Lustre: Skipped 2 previous similar messages [ 3985.016115] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.1/24@tcp [ 3985.028683] Lustre: Skipped 1 previous similar message [ 3992.042904] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3992.049090] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3992.070525] LustreError: Skipped 1 previous similar message [ 3992.150228] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3992.548766] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3993.241217] LustreError: 117965:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3993.566116] Lustre: server umount lustre-OST0000 complete [ 4001.250944] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4001.259051] LustreError: Skipped 4 previous similar messages [ 4001.261088] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4001.275183] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4001.284422] Lustre: Skipped 1 previous similar message [ 4002.786489] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4004.462468] LustreError: 118166:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4004.468356] LustreError: 118166:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4004.598043] Lustre: server umount lustre-MDT0000 complete [ 4007.910849] LustreError: 114412:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4007.941057] LustreError: 114412:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4008.324576] LustreError: 114396:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768558252 with bad export cookie 7759693959111656247 [ 4008.335583] LustreError: 114396:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4008.337422] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4008.511518] LustreError: 118367:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 4008.523968] LustreError: 118367:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4008.709440] Lustre: server umount lustre-MDT0001 complete [ 4019.959797] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [ 4022.548713] Key type lgssc unregistered [ 4023.036625] LNet: 118899:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4023.051834] LNetError: 118899:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4024.102574] LNet: Removed LNI 192.168.203.101@tcp [ 4025.459107] Key type .llcrypt unregistered [ 4025.463147] Key type ._llcrypt unregistered [ 4058.137309] Key type ._llcrypt registered [ 4058.141941] Key type .llcrypt registered [ 4058.264225] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 4059.417805] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4059.718915] alg: No test for adler32 (adler32-zlib) [ 4060.925810] Lustre: Lustre: Build Version: 2.17.50_30_gc0571ea [ 4061.200251] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [ 4062.904903] Key type lgssc registered [ 4064.057948] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4073.062591] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4073.075302] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4074.643479] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 4074.686787] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.203.1/24@tcp [ 4074.694424] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4079.834907] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 4084.196595] LustreError: 120444:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4084.227438] LustreError: 120444:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4089.327127] LustreError: 120444:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4089.719171] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4090.119919] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 4090.171105] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.1/24@tcp [ 4090.208396] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4094.373990] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 4100.040402] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4104.665131] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4112.242392] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4112.523479] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4118.508612] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 4119.747670] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:201 to 0x280000400:225) [ 4124.846715] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4140.513734] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4140.518403] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4140.540919] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4140.546754] Lustre: Skipped 1 previous similar message [ 4142.714581] LustreError: 122737:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4143.174383] Lustre: server umount lustre-OST0000 complete [ 4150.754370] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4150.780749] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4150.786089] Lustre: Skipped 2 previous similar messages [ 4153.287587] LustreError: 122940:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4153.299499] LustreError: 122940:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4153.548495] Lustre: server umount lustre-MDT0000 complete [ 4155.876624] LustreError: 120440:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4155.908430] LustreError: 120440:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4157.785930] LustreError: 120423:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768558401 with bad export cookie 16476806565766680659 [ 4157.803509] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4157.808489] LustreError: 120423:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4157.956809] LustreError: 123142:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 4157.971911] LustreError: 123142:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4158.211868] Lustre: server umount lustre-MDT0001 complete [ 4168.957601] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [ 4172.096814] Key type lgssc unregistered [ 4172.496327] LNet: 123675:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4172.506028] LNetError: 123675:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4172.521922] LNet: Removed LNI 192.168.203.101@tcp [ 4173.553112] Key type .llcrypt unregistered [ 4173.557455] Key type ._llcrypt unregistered [ 4202.331930] Key type ._llcrypt registered [ 4202.334696] Key type .llcrypt registered [ 4202.466754] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_hostid [ 4216.385075] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 4217.585780] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4217.674265] alg: No test for adler32 (adler32-zlib) [ 4218.750911] Lustre: Lustre: Build Version: 2.17.50_30_gc0571ea [ 4218.979786] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [ 4220.736164] Key type lgssc registered [ 4221.861456] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4261.285788] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4261.337021] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4262.721938] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4262.811683] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4262.945721] Lustre: lustre-MDT0000: new disk, initializing [ 4263.079491] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4263.106699] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4269.456553] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 4282.306031] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4282.524626] Lustre: 127041:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4282.657910] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4282.666807] Lustre: Skipped 1 previous similar message [ 4282.812583] Lustre: lustre-MDT0001: new disk, initializing [ 4282.981484] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4283.051555] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4283.071163] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4287.526975] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 4296.727994] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4300.832184] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4309.349541] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4309.744551] Lustre: lustre-OST0000: new disk, initializing [ 4309.754494] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4309.859701] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4315.309362] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 4317.781939] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4317.793306] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4317.895208] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4323.211289] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4327.644880] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4327.880666] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4332.799540] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4333.067785] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4338.676971] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4338.683316] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4338.687061] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4338.691962] Lustre: Skipped 1 previous similar message [ 4338.712469] LustreError: Skipped 1 previous similar message [ 4341.376743] LustreError: 129975:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4341.714285] Lustre: server umount lustre-OST0000 complete [ 4348.898613] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4348.902333] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4348.920407] Lustre: Skipped 1 previous similar message [ 4348.934585] Lustre: Skipped 2 previous similar messages [ 4354.022618] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4354.028332] Lustre: Skipped 1 previous similar message [ 4358.122627] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4358.132090] Lustre: Skipped 1 previous similar message [ 4360.160326] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4360.293444] LustreError: 130177:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4360.304703] LustreError: 130177:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4360.440870] Lustre: server umount lustre-MDT0000 complete [ 4364.057794] LustreError: 127033:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768558608 with bad export cookie 13418289873874839236 [ 4364.064446] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4364.075949] LustreError: 127033:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4364.243123] LustreError: 130380:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4364.251418] LustreError: 130380:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4364.482678] Lustre: server umount lustre-MDT0001 complete [ 4372.796576] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4373.252555] LustreError: 130856:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4373.278933] LustreError: 130856:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4373.368520] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4378.615515] LustreError: 130856:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4378.920670] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 4383.715614] LustreError: 130856:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4389.036142] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4389.341842] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4394.624315] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 4402.003265] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4406.240394] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4413.595371] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4413.985553] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4421.263458] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 4427.300410] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4433.535417] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4433.989276] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4439.420673] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4439.734145] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4452.701518] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4453.114100] Lustre: lustre-OST0001: new disk, initializing [ 4453.123402] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4453.321668] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4461.762348] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4461.777377] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4462.053916] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4462.208258] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 4471.307943] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4471.554700] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4476.079937] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4476.319863] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4480.311520] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8b2646309000.ost_server_uuid 50 [ 4481.868552] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8b2646309000.ost_server_uuid in IDLE state after 0 sec [ 4492.769982] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4492.780448] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4492.790030] LustreError: Skipped 1 previous similar message [ 4492.798449] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4492.837840] Lustre: Skipped 1 previous similar message [ 4498.058868] LustreError: 135100:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 4498.061832] LustreError: 135100:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4498.143959] Lustre: server umount lustre-OST0001 complete [ 4501.991602] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4501.995935] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4502.004352] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4502.008180] Lustre: Skipped 3 previous similar messages [ 4503.028290] LustreError: 132451:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4503.040588] LustreError: 132451:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4508.136306] LustreError: 132448:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4508.154594] LustreError: 132448:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4516.320283] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4516.401123] LustreError: 135302:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4516.404214] LustreError: 135302:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4516.688383] Lustre: server umount lustre-OST0000 complete [ 4522.470363] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4522.480460] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4522.492197] Lustre: Skipped 1 previous similar message [ 4522.499929] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4522.506019] Lustre: Skipped 6 previous similar messages [ 4527.341723] LustreError: 135506:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 4527.344428] LustreError: 135506:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4527.525411] Lustre: server umount lustre-MDT0000 complete [ 4528.620050] LustreError: 130852:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4528.634480] LustreError: 130852:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4532.111683] LustreError: 130835:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768558776 with bad export cookie 13418289873874840118 [ 4532.116807] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4532.122882] LustreError: 130835:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4532.591620] Lustre: server umount lustre-MDT0001 complete [ 4555.582501] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [ 4559.402671] Key type lgssc unregistered [ 4559.726227] LNet: 136700:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4559.738294] LNetError: 136700:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4559.757218] LNet: Removed LNI 192.168.203.101@tcp [ 4560.769445] Key type .llcrypt unregistered [ 4560.777334] Key type ._llcrypt unregistered [ 4611.280744] Key type ._llcrypt registered [ 4611.284746] Key type .llcrypt registered [ 4611.445496] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_hostid [ 4627.319940] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 4629.136335] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4629.202732] alg: No test for adler32 (adler32-zlib) [ 4630.348662] Lustre: Lustre: Build Version: 2.17.50_30_gc0571ea [ 4630.661880] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [ 4632.384194] Key type lgssc registered [ 4634.012507] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4669.819938] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4669.898000] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4671.159363] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4671.173526] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4671.256505] Lustre: lustre-MDT0000: new disk, initializing [ 4671.387108] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4671.412726] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4676.205418] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 4689.284929] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4689.513621] Lustre: 140821:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4689.656056] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4689.668834] Lustre: Skipped 1 previous similar message [ 4689.778562] Lustre: lustre-MDT0001: new disk, initializing [ 4689.917584] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4689.959844] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4689.973172] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4695.468226] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 4703.823882] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4708.416557] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4715.918027] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4716.224873] Lustre: lustre-OST0000: new disk, initializing [ 4716.228644] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4716.385054] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4723.063353] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 4724.335128] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4724.349831] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4724.463936] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4731.956871] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4736.212386] Lustre: Failing over lustre-OST0000 [ 4736.265408] LustreError: 143254:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4736.355359] Lustre: server umount lustre-OST0000 complete [ 4754.400602] Lustre: 139044:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1768558982/real 1768558982] req@0000000078ab7d9b x1854468389896832/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1768558998 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4754.442220] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4756.294707] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4756.499338] Lustre: 139041:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1768558984/real 1768558984] req@00000000a19cd3c8 x1854468389897088/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1768559000 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4756.549321] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4756.853482] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4756.883851] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4759.858798] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4761.680281] Lustre: 139041:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1768558989/real 1768558989] req@00000000d93a0750 x1854468389899136/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1768559005 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4761.698563] Lustre: 139041:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 4762.731727] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 4764.436863] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4764.444943] Lustre: lustre-OST0000: Recovery over after 0:05, of 3 clients 3 recovered and 0 were evicted. [ 4766.230311] Lustre: Failing over lustre-MDT0000 [ 4766.396092] LustreError: 144226:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4766.400752] LustreError: 144226:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4766.587197] Lustre: server umount lustre-MDT0000 complete [ 4766.690338] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4766.706149] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4766.726716] LustreError: 140832:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4766.737468] LustreError: 140832:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4767.264471] Lustre: 139041:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1768558994/real 1768558994] req@00000000f35dc742 x1854468389900160/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1768559010 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4767.311982] Lustre: 139041:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4769.250064] LustreError: 140832:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4769.257281] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4769.280389] LustreError: 140832:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4769.311024] Lustre: Skipped 1 previous similar message [ 4774.374244] LustreError: 140832:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4774.387885] LustreError: 140832:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 4779.490600] LustreError: 140832:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4779.506287] LustreError: 140832:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4782.054954] Lustre: 140823:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1768559010/real 1768559010] req@0000000029f74ecc x1854468389909248/t0(0) o101->MGC192.168.203.101@tcp@0@lo:26/25 lens 328/344 e 0 to 1 dl 1768559026 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4782.093775] Lustre: 140823:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4782.108410] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4782.130092] Lustre: 140823:0:(mgc_request.c:1910:mgc_process_log()) MGC192.168.203.101@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 4784.865533] LustreError: 140828:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.1@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4784.877600] LustreError: 140828:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 4786.728658] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4792.830552] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4792.918879] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4796.127298] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4797.936046] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4797.939335] Lustre: Skipped 1 previous similar message [ 4797.991830] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 4798.071308] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4798.662938] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 4810.735804] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4810.741369] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4810.754060] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4811.917254] LustreError: 145112:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4811.930051] LustreError: 145112:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4812.290950] Lustre: server umount lustre-OST0000 complete [ 4818.401292] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4818.406484] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4823.522363] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4823.529256] Lustre: Skipped 2 previous similar messages [ 4827.621152] LustreError: 145313:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 4827.633748] LustreError: 145313:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4827.705764] Lustre: server umount lustre-MDT0000 complete [ 4828.645353] LustreError: 140832:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4828.669228] LustreError: 140832:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 10 previous similar messages [ 4832.213250] LustreError: 140810:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768559076 with bad export cookie 15236093909621197288 [ 4832.216041] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4832.224080] LustreError: 140810:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4832.455855] LustreError: 145516:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4832.467709] LustreError: 145516:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4832.893941] Lustre: server umount lustre-MDT0001 complete [ 4845.876148] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [ 4849.437641] Key type lgssc unregistered [ 4849.834699] LNet: 146048:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4849.849075] LNetError: 146048:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4849.890870] LNet: Removed LNI 192.168.203.101@tcp [ 4851.016618] Key type .llcrypt unregistered [ 4851.022601] Key type ._llcrypt unregistered [ 4888.799533] Key type ._llcrypt registered [ 4888.805487] Key type .llcrypt registered [ 4888.897442] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 4890.163248] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4890.215048] alg: No test for adler32 (adler32-zlib) [ 4891.214487] Lustre: Lustre: Build Version: 2.17.50_30_gc0571ea [ 4891.406886] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [ 4893.064199] Key type lgssc registered [ 4894.842767] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4903.997994] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4904.021961] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4905.786712] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4911.029045] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 4915.176066] LustreError: 147550:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4915.195896] LustreError: 147550:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4919.388105] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4920.308352] LustreError: 147550:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4923.574204] LustreError: 147579:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4923.591928] LustreError: 147579:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4923.600074] LustreError: 147579:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 18, retries 0, failed: rc = -5 [ 4923.857605] LustreError: 148103:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4924.479697] Lustre: server umount lustre-MDT0000 complete [ 4942.283346] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_hostid [ 4956.145827] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 5000.529067] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5000.842630] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5000.876408] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5000.958993] Lustre: lustre-MDT0000: new disk, initializing [ 5001.059529] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5001.080659] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5005.131911] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 5015.201606] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5015.319981] Lustre: 150737:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5015.373738] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5015.376738] Lustre: Skipped 1 previous similar message [ 5015.467123] Lustre: lustre-MDT0001: new disk, initializing [ 5015.583869] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5015.654135] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5015.672381] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5020.250065] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 5028.715162] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5034.973811] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5044.981693] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5045.525580] Lustre: lustre-OST0000: new disk, initializing [ 5045.541923] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5045.678266] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5052.049689] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5052.063842] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5052.287125] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5053.365989] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 5061.076934] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5066.208561] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 5068.214933] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 05:28:30 (1768559310) [ 5313.505721] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5313.509192] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5313.520188] Lustre: Skipped 1 previous similar message [ 5313.535289] Lustre: Skipped 2 previous similar messages [ 5318.632715] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5318.644503] Lustre: Skipped 2 previous similar messages [ 5319.011733] LustreError: 154257:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5319.024312] LustreError: 154257:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5319.426364] Lustre: server umount lustre-MDT0000 complete [ 5323.617320] LustreError: 150730:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768559567 with bad export cookie 17859788371154439108 [ 5323.628838] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5323.635208] LustreError: 150730:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5323.745322] LustreError: 154133:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5323.749076] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5323.780361] Lustre: Skipped 1 previous similar message [ 5323.785529] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5328.867091] LustreError: 154133:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5328.867131] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5329.878841] LustreError: 154460:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5329.893337] LustreError: 154460:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5330.245829] Lustre: server umount lustre-MDT0001 complete [ 5348.832924] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5348.860293] LustreError: 154664:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 5348.872254] LustreError: 154664:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5349.149806] Lustre: server umount lustre-OST0000 complete [ 5359.792840] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_hostid [ 5369.272519] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 5408.940339] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5409.685745] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5409.817889] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5410.015493] Lustre: lustre-MDT0000: new disk, initializing [ 5410.225308] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5410.265657] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5415.490259] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 5430.109197] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5430.258346] Lustre: 156995:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5430.382622] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5430.397493] Lustre: Skipped 1 previous similar message [ 5430.689438] Lustre: lustre-MDT0001: new disk, initializing [ 5430.935346] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5431.073508] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5431.098210] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5438.071537] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 5448.143298] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5453.105923] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5462.004229] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5462.512644] Lustre: lustre-OST0000: new disk, initializing [ 5462.517771] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5462.629199] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5468.767449] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5468.781276] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5468.962564] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5470.582957] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 5480.100910] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5486.018656] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5486.341263] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5491.898988] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5492.188267] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5494.755376] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5494.764507] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5494.788333] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5494.798022] Lustre: Skipped 1 previous similar message [ 5500.604595] LustreError: 159927:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5500.612618] LustreError: 159927:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5500.937375] Lustre: server umount lustre-OST0000 complete [ 5507.559980] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5507.575486] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5507.590137] Lustre: Skipped 1 previous similar message [ 5507.603120] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5507.613485] Lustre: Skipped 2 previous similar messages [ 5511.751722] LustreError: 160129:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5511.760527] LustreError: 160129:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5511.976364] Lustre: server umount lustre-MDT0000 complete [ 5515.238899] LustreError: 157006:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5515.273038] LustreError: 157006:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5516.370923] LustreError: 156985:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768559760 with bad export cookie 17859788371154597938 [ 5516.385059] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5516.385128] LustreError: 156985:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 5516.755157] Lustre: server umount lustre-MDT0001 complete [ 5527.981618] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 05:36:09 (1768559769) [ 5548.846516] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_hostid [ 5559.648542] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 5603.358658] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5603.721648] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5603.746536] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5603.817346] Lustre: lustre-MDT0000: new disk, initializing [ 5603.971229] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5604.024628] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5609.501708] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 5622.206685] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5622.368892] Lustre: 163314:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5622.422667] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 5622.424332] Lustre: Skipped 2 previous similar messages [ 5622.626845] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5628.222470] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 5636.907118] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5641.187857] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5649.088374] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5649.340193] Lustre: lustre-OST0000: new disk, initializing [ 5649.343023] Lustre: Skipped 1 previous similar message [ 5649.350476] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5649.355300] Lustre: Skipped 2 previous similar messages [ 5656.629918] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 5658.681484] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5658.693434] Lustre: Skipped 1 previous similar message [ 5658.699780] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5658.820864] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5664.775218] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5674.466897] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5674.469830] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5674.472697] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5674.482593] LustreError: Skipped 1 previous similar message [ 5674.493648] Lustre: Skipped 2 previous similar messages [ 5678.468497] LustreError: 165860:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5678.479177] LustreError: 165860:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 5678.949695] Lustre: server umount lustre-OST0000 complete [ 5684.196981] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5684.205249] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5690.610870] Lustre: server umount lustre-MDT0000 complete [ 5693.922754] LustreError: 164104:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5693.952885] LustreError: 164104:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5694.894933] LustreError: 163307:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768559939 with bad export cookie 17859788371154598813 [ 5694.902677] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5694.907586] LustreError: 163307:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5695.313899] Lustre: server umount lustre-MDT0001 complete [ 5704.777510] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 05:39:07 (1768559947) [ 5722.467727] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_hostid [ 5732.822650] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 5778.542238] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5778.983996] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5779.036877] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5779.122321] Lustre: lustre-MDT0000: new disk, initializing [ 5779.205506] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5779.214713] Lustre: Skipped 2 previous similar messages [ 5779.240251] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5784.865611] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 5797.255724] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5797.438108] Lustre: 169245:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5797.723646] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5803.260838] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 5812.264912] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5817.271576] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5824.885678] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5831.783979] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5831.919239] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5832.539370] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 5840.949888] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5852.647254] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5852.649700] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5852.687898] Lustre: Skipped 1 previous similar message [ 5852.696131] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5852.696140] Lustre: Skipped 5 previous similar messages [ 5855.744349] LustreError: 171785:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5855.747256] LustreError: 171785:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 5855.993906] Lustre: server umount lustre-OST0000 complete [ 5866.414908] Lustre: server umount lustre-MDT0000 complete [ 5868.003383] LustreError: 169251:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5868.012880] LustreError: 169251:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5870.772284] LustreError: 169237:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768560114 with bad export cookie 17859788371154599905 [ 5870.799152] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5871.160466] Lustre: server umount lustre-MDT0001 complete [ 5888.988913] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [ 5893.462539] Key type lgssc unregistered [ 5894.007729] LNet: 172749:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5894.025954] LNetError: 172749:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5894.044844] LNet: Removed LNI 192.168.203.101@tcp [ 5895.396639] Key type .llcrypt unregistered [ 5895.399877] Key type ._llcrypt unregistered [ 5926.932984] Key type ._llcrypt registered [ 5926.938570] Key type .llcrypt registered [ 5927.044678] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 5928.204310] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5928.212575] alg: No test for adler32 (adler32-zlib) [ 5929.489771] Lustre: Lustre: Build Version: 2.17.50_30_gc0571ea [ 5929.708361] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [ 5931.496617] Key type lgssc registered [ 5932.825843] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5942.104705] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5942.128619] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5943.969318] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5950.460040] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 5953.525799] LustreError: 174148:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5953.543526] LustreError: 174148:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5958.627508] LustreError: 174148:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5962.667147] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5963.136190] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5969.473111] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 5976.951740] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5982.131443] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5990.762502] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5991.255374] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5998.686965] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 6005.531482] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6013.836882] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 6039.522318] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6039.534877] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6039.550591] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6042.758869] LustreError: 176416:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6043.115937] Lustre: server umount lustre-OST0000 complete [ 6049.763472] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6049.774115] Lustre: Skipped 1 previous similar message [ 6049.777549] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6049.784767] Lustre: Skipped 1 previous similar message [ 6054.072861] LustreError: 176618:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6054.109483] LustreError: 176618:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6054.692775] Lustre: server umount lustre-MDT0000 complete [ 6054.881838] LustreError: 174148:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6054.897190] LustreError: 174148:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6059.000049] LustreError: 174144:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6059.028974] LustreError: 174144:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6059.158296] LustreError: 174129:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768560303 with bad export cookie 2538682954093663798 [ 6059.170531] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6059.299473] LustreError: 176821:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6059.316379] LustreError: 176821:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6059.613422] Lustre: server umount lustre-MDT0001 complete [ 6072.917420] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [ 6076.444621] Key type lgssc unregistered [ 6076.820778] LNet: 177354:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6076.826960] LNetError: 177354:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6077.868861] LNet: Removed LNI 192.168.203.101@tcp [ 6079.073347] Key type .llcrypt unregistered [ 6079.080552] Key type ._llcrypt unregistered [ 6109.445880] Key type ._llcrypt registered [ 6109.449534] Key type .llcrypt registered [ 6109.598850] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 6110.761161] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6110.861793] alg: No test for adler32 (adler32-zlib) [ 6112.152515] Lustre: Lustre: Build Version: 2.17.50_30_gc0571ea [ 6112.364271] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [ 6114.008199] Key type lgssc registered [ 6115.260139] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6124.180757] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6124.213059] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6125.894346] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6130.296781] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 6135.287132] LustreError: 178750:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6135.322150] LustreError: 178750:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6140.390648] LustreError: 178750:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6141.869307] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6142.130199] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6148.788715] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 6156.694201] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6162.558586] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6171.627893] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6172.057767] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6178.351789] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 6185.040655] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6195.666714] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6200.806203] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6200.810186] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6200.823908] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6205.923483] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6205.929498] Lustre: Skipped 3 previous similar messages [ 6211.044246] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6211.063122] Lustre: Skipped 2 previous similar messages [ 6215.136176] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6215.172725] LustreError: 181016:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6215.434918] Lustre: server umount lustre-OST0000 complete [ 6222.972688] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff8b2652084000.ost_server_uuid 50 [ 6224.769491] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 6239.714456] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6239.723057] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6239.729080] Lustre: Skipped 1 previous similar message [ 6239.733317] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6239.741122] Lustre: Skipped 1 previous similar message [ 6241.766322] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6241.919466] LustreError: 181384:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6241.925907] LustreError: 181384:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6242.073901] Lustre: server umount lustre-MDT0000 complete [ 6245.856182] LustreError: 178731:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768560489 with bad export cookie 7210708626270962017 [ 6245.866967] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6245.870356] LustreError: 178731:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6246.010081] LustreError: 181585:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6246.014517] LustreError: 181585:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6246.343422] Lustre: server umount lustre-MDT0001 complete [ 6256.531874] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 05:48:18 (1768560498) [ 6263.517760] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6263.901650] LustreError: 182258:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6264.023266] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6268.682178] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 6269.409717] LustreError: 182258:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6274.534514] LustreError: 182258:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6279.329916] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6279.649407] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 6279.660944] Lustre: Skipped 2 previous similar messages [ 6279.881173] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6284.335417] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 6290.666921] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6295.332753] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6302.659039] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6303.213539] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6309.372250] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6310.049874] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 6316.775642] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6323.766400] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6324.001786] Lustre: lustre-OST0001: new disk, initializing [ 6324.009810] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6324.160430] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6330.713168] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 6333.542804] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6333.558662] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6333.699731] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6339.368599] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6344.163748] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6344.210045] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6344.219819] Lustre: Skipped 1 previous similar message [ 6348.357112] LustreError: 185457:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6348.361671] LustreError: 185457:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6348.481742] Lustre: server umount lustre-OST0000 complete [ 6349.295601] LustreError: 183855:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6349.316631] LustreError: 183855:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6354.594813] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6354.817564] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6359.468603] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6359.723707] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6365.922194] LustreError: 183852:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.1@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6365.936499] LustreError: 183852:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 6374.881191] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6374.888480] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6374.893972] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6374.901204] Lustre: Skipped 2 previous similar messages [ 6374.920764] Lustre: Skipped 2 previous similar messages [ 6380.288398] LustreError: 186181:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 6380.292459] LustreError: 186181:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6380.470729] Lustre: server umount lustre-OST0001 complete [ 6387.169081] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6387.174033] LustreError: Skipped 1 previous similar message [ 6387.183079] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6390.996384] LustreError: 186383:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 6391.010867] LustreError: 186383:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6391.403582] Lustre: server umount lustre-MDT0000 complete [ 6395.110948] LustreError: 182239:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768560639 with bad export cookie 7210708626270963088 [ 6395.124638] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6395.130381] LustreError: 182239:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6395.524895] Lustre: server umount lustre-MDT0001 complete [ 6419.857880] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 05:51:02 (1768560662) [ 6426.882354] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6427.093930] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6427.109249] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6427.245646] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6431.974828] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 6441.439345] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6441.527699] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6441.556785] Lustre: 188052:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6447.112932] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 6455.724367] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6460.530986] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6468.118389] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6468.277883] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6476.160685] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 6477.504435] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6483.154800] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6491.363904] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6491.523391] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6491.905048] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6491.921093] Lustre: Skipped 2 previous similar messages [ 6499.826832] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 6507.114223] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6511.595567] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6511.599379] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6511.613067] Lustre: Skipped 2 previous similar messages [ 6511.632408] Lustre: Skipped 6 previous similar messages [ 6516.491750] LustreError: 191245:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6516.505831] LustreError: 191245:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 6516.721928] LustreError: 189680:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6516.742236] LustreError: 189680:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 10 previous similar messages [ 6516.796562] Lustre: server umount lustre-OST0000 complete [ 6536.673846] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6539.390949] Lustre: server umount lustre-OST0001 complete [ 6544.353590] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6544.366676] Lustre: Skipped 2 previous similar messages [ 6549.697505] LustreError: 191673:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 6549.709378] LustreError: 191673:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6549.829672] Lustre: server umount lustre-MDT0000 complete [ 6554.638697] LustreError: 190603:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768560798 with bad export cookie 7210708626270964495 [ 6554.644507] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6554.671756] LustreError: 190603:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6555.223644] Lustre: server umount lustre-MDT0001 complete [ 6580.684106] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 05:53:43 (1768560823) [ 6598.956802] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_hostid [ 6610.029814] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 6654.406280] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6654.598295] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6654.628695] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6654.679903] Lustre: lustre-MDT0000: new disk, initializing [ 6654.754918] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6654.770028] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6658.894422] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 6669.442992] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6669.534522] Lustre: 195667:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6669.599158] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6669.612186] Lustre: Skipped 1 previous similar message [ 6669.685566] Lustre: lustre-MDT0001: new disk, initializing [ 6669.789692] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6669.802690] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6673.544446] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 6682.031374] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6688.100818] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6698.686841] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6699.329671] Lustre: lustre-OST0000: new disk, initializing [ 6699.350379] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6705.881691] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6705.895166] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6706.068039] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6708.099487] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 6717.100824] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6722.463434] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6722.750283] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6728.294823] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6728.630727] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6731.757092] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6731.757842] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6731.787592] Lustre: Skipped 1 previous similar message [ 6731.797993] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6731.805188] LustreError: Skipped 2 previous similar messages [ 6731.837521] Lustre: Skipped 8 previous similar messages [ 6737.345729] LustreError: 198602:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6737.359442] LustreError: 198602:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 6737.519243] Lustre: server umount lustre-OST0000 complete [ 6749.018894] Lustre: server umount lustre-MDT0000 complete [ 6752.234300] LustreError: 195675:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6752.260383] LustreError: 195675:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 14 previous similar messages [ 6753.404265] LustreError: 195659:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768560997 with bad export cookie 7210708626270966056 [ 6753.408118] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6753.417109] LustreError: 195659:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6754.005239] Lustre: server umount lustre-MDT0001 complete [ 6763.558397] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6769.105193] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 6778.811357] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6784.133511] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 6789.790791] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6794.755753] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6803.784560] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6813.773604] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 6819.758964] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6823.947679] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6824.246427] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6828.904416] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6829.328386] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6834.145973] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6834.169243] LustreError: Skipped 1 previous similar message [ 6845.408295] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6845.544425] Lustre: server umount lustre-OST0000 complete [ 6853.559848] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6859.201896] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6888.664194] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6895.230513] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 6899.239471] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6899.250038] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6899.263768] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6899.281319] Lustre: Skipped 1 previous similar message [ 6900.993950] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6907.517825] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 4 sec [ 6914.530866] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6914.541735] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6914.547874] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6914.564064] Lustre: Skipped 6 previous similar messages [ 6914.587072] Lustre: Skipped 11 previous similar messages [ 6918.222115] LustreError: 203861:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6918.226025] LustreError: 203861:0:(obd_class.h:479:obd_check_dev()) Skipped 21 previous similar messages [ 6918.631542] Lustre: server umount lustre-OST0000 complete [ 6928.500532] Lustre: server umount lustre-MDT0000 complete [ 6929.896627] LustreError: 200903:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6929.908993] LustreError: 200903:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 6932.385534] LustreError: 199461:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768561176 with bad export cookie 7210708626270966938 [ 6932.406474] LustreError: 199461:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6932.414932] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6933.302542] Lustre: server umount lustre-MDT0001 complete [ 6948.681606] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 05:59:50 (1768561190) [ 6957.459810] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6958.029905] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6958.045153] Lustre: Skipped 6 previous similar messages [ 6962.509325] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 6971.372277] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6976.024541] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 6983.009201] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6987.465375] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6996.639868] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7004.437985] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 7010.204337] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7014.597184] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7014.831454] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7018.600400] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7018.779973] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7025.890737] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7026.125520] Lustre: lustre-OST0001: new disk, initializing [ 7026.136447] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7031.924298] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7031.938604] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7032.044616] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7032.542147] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 7040.377533] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7045.495749] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7045.719849] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7052.217905] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7052.487344] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7057.899225] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7057.917846] LustreError: Skipped 3 previous similar messages [ 7069.157458] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7069.322750] Lustre: server umount lustre-OST0001 complete [ 7076.525651] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7076.810402] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 7082.460058] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7082.709775] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 7112.861804] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7118.342596] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7118.378074] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7118.382038] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7118.412378] Lustre: Skipped 1 previous similar message [ 7119.552838] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 7126.151705] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7128.171739] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7136.729426] Lustre: server umount lustre-OST0001 complete [ 7160.505962] Lustre: server umount lustre-OST0000 complete [ 7170.696265] Lustre: server umount lustre-MDT0000 complete [ 7174.138302] LustreError: 204921:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768561418 with bad export cookie 7210708626270967904 [ 7174.141385] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7174.143901] LustreError: 204921:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7180.423577] LustreError: 211438:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 7180.441424] LustreError: 211438:0:(obd_class.h:479:obd_check_dev()) Skipped 37 previous similar messages [ 7180.664358] Lustre: server umount lustre-MDT0001 complete [ 7207.330682] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 06:04:09 (1768561449) [ 7215.622446] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7216.125919] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7216.176621] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7223.519868] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 7233.242387] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7233.351124] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7233.391489] Lustre: 212907:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7239.028729] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 7245.506178] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7250.132759] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7259.141759] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7259.349833] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7266.105250] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 7271.546854] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7281.421330] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7281.501926] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7288.458448] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 7294.305859] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7296.279303] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7301.410821] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7301.782875] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7306.560712] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7306.778145] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7311.728709] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8b2647edf800.ost_server_uuid 50 [ 7314.083677] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8b2647edf800.ost_server_uuid in IDLE state after 0 sec [ 7315.569793] Lustre: Permanently deactivating lustre-OST0001 [ 7315.605657] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7320.265831] Lustre: Permanently reactivating lustre-OST0001 [ 7320.288964] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 7320.294412] Lustre: Skipped 2 previous similar messages [ 7325.665938] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7325.683028] LustreError: Skipped 4 previous similar messages [ 7325.688469] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7325.699886] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7325.706758] Lustre: Skipped 11 previous similar messages [ 7325.728837] Lustre: Skipped 20 previous similar messages [ 7331.215886] Lustre: server umount lustre-OST0001 complete [ 7335.908034] LustreError: 214826:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7335.936945] LustreError: 214826:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 37 previous similar messages [ 7341.436740] Lustre: server umount lustre-OST0000 complete [ 7351.662562] Lustre: server umount lustre-MDT0000 complete [ 7355.722419] LustreError: 213634:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768561599 with bad export cookie 7210708626270969192 [ 7355.743428] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7355.748393] LustreError: 213634:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7356.080569] Lustre: server umount lustre-MDT0001 complete [ 7382.753865] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 06:07:04 (1768561624) [ 7390.593434] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7390.784183] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7390.809698] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7390.818951] Lustre: Skipped 2 previous similar messages [ 7395.797712] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 7405.080753] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7405.176727] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7405.196292] Lustre: 219021:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7410.478613] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 7417.118936] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7421.213426] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7429.656144] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7429.964594] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7429.993418] Lustre: Permanently deactivating lustre-OST0000 [ 7430.006217] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7437.231667] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 7438.374732] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7443.056198] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7450.824721] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7450.902284] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7460.228476] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 7461.380768] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7467.935677] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7470.844672] Lustre: Permanently reactivating lustre-OST0000 [ 7470.868508] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7470.876442] Lustre: Skipped 2 previous similar messages [ 7476.349050] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 7476.384757] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 38 seconds. I think it's dead, and I am evicting it. exp 00000000b0caa50e, cur 1768561720 deadline 1768561712 last 1768561682 [ 7476.494537] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7501.280321] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7501.436250] Lustre: server umount lustre-OST0001 complete [ 7511.294701] Lustre: server umount lustre-OST0000 complete [ 7521.647705] Lustre: server umount lustre-MDT0000 complete [ 7525.957445] LustreError: 219014:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768561770 with bad export cookie 7210708626270970865 [ 7525.962257] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7525.977407] LustreError: 219014:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 7526.402921] Lustre: server umount lustre-MDT0001 complete [ 7537.705368] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [ 7540.596669] Key type lgssc unregistered [ 7540.924980] LNet: 223433:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7540.931243] LNetError: 223433:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7540.949116] LNet: Removed LNI 192.168.203.101@tcp [ 7541.901314] Key type .llcrypt unregistered [ 7541.904662] Key type ._llcrypt unregistered [ 7566.125379] Key type ._llcrypt registered [ 7566.134039] Key type .llcrypt registered [ 7566.299614] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 7567.497281] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7567.565641] alg: No test for adler32 (adler32-zlib) [ 7568.684891] Lustre: Lustre: Build Version: 2.17.50_30_gc0571ea [ 7568.891193] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [ 7570.585325] Key type lgssc registered [ 7571.963579] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7582.738645] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7582.759833] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7584.353326] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7589.570276] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 7594.480165] LustreError: 224881:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7594.497804] LustreError: 224881:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7598.968783] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7599.089209] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7599.116670] Lustre: 224871:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7599.130163] Lustre: Permanently deactivating lustre-MDT0001 [ 7599.144562] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7599.473199] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7599.761985] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7599.769788] LustreError: 224912:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 7599.793989] LustreError: 224912:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 7599.818596] LustreError: 224912:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 15, retries 1, failed: rc = -108 [ 7604.195994] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 7610.927352] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7614.960660] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7622.722578] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7623.011695] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7624.057675] LustreError: 226570:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7624.093636] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 7629.225872] LustreError: 226496:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7629.244112] LustreError: 226496:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7629.262916] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7630.400134] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 7634.422760] LustreError: 226495:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7634.446105] LustreError: 226495:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7637.596072] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7639.523269] LustreError: 226497:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7639.544996] LustreError: 226497:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7645.770089] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7645.972814] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7651.323817] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7651.325564] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7651.838575] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 7657.240475] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7661.333335] Lustre: Permanently reactivating lustre-MDT0001 [ 7661.365979] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7661.377765] Lustre: Skipped 2 previous similar messages [ 7664.583163] Lustre: lustre-MDT0001: Client 4657ad02-cb8d-4a81-bcb3-2581eef77b43 (at 192.168.203.1@tcp) reconnecting [ 7668.839834] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7670.692815] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7677.018731] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7677.233121] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7681.938350] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7682.105134] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7684.952700] Lustre: Permanently deactivating lustre-MDT0001 [ 7684.976191] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7684.989561] Lustre: Skipped 2 previous similar messages [ 7692.908309] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7701.988497] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7702.000570] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7702.017990] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7702.509862] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7702.527736] Lustre: Skipped 2 previous similar messages [ 7702.534705] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7702.544162] Lustre: Skipped 3 previous similar messages [ 7704.902405] LustreError: 228947:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 7705.055550] Lustre: server umount lustre-MDT0000 complete [ 7707.620423] LustreError: 224881:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7707.648533] LustreError: 224881:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 7709.340742] LustreError: 224861:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768561953 with bad export cookie 5686631367233904020 [ 7709.345160] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7709.356402] LustreError: 224861:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7709.527224] LustreError: 229148:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 7709.540160] LustreError: 229148:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 7709.834361] Lustre: server umount lustre-MDT0001 complete [ 7715.466794] LustreError: 229350:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 7715.469580] LustreError: 229350:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 7715.551988] Lustre: server umount lustre-OST0000 complete [ 7727.854984] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 06:12:50 (1768561970) [ 7739.435555] LustreError: 230357:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 7739.439174] LustreError: 230357:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 7739.646656] Lustre: server umount lustre-OST0001 complete [ 7747.329504] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_hostid [ 7757.682684] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 7797.455139] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7797.753917] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7797.761698] Lustre: Skipped 2 previous similar messages [ 7797.785674] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7797.866805] Lustre: lustre-MDT0000: new disk, initializing [ 7798.001237] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7798.026611] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7803.091801] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 7814.515234] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7814.598912] Lustre: 232533:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7814.657891] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7814.664506] Lustre: Skipped 1 previous similar message [ 7814.794533] Lustre: lustre-MDT0001: new disk, initializing [ 7814.884475] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7814.926277] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7814.944231] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7819.543552] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 7828.328366] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7832.836120] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7840.006939] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7840.277895] Lustre: lustre-OST0000: new disk, initializing [ 7840.283567] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7840.365551] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7846.259482] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 7850.590288] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7850.605323] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7850.733767] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7854.883256] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7857.165454] LustreError: 232540:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7866.101134] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7866.268517] Lustre: lustre-OST0001: new disk, initializing [ 7866.271285] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7867.253276] LustreError: 232540:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7867.276073] LustreError: 232540:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7871.704376] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 7872.035737] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7872.047411] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7872.114117] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7877.280119] LustreError: 232540:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7878.728985] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7880.027608] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7882.212676] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7882.213813] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7882.217591] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7882.231234] Lustre: Skipped 1 previous similar message [ 7882.254132] LustreError: Skipped 1 previous similar message [ 7887.336369] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7887.357669] Lustre: Skipped 3 previous similar messages [ 7887.676138] LustreError: 236015:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 7887.679601] LustreError: 236015:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7887.906714] Lustre: server umount lustre-OST0001 complete [ 7888.645719] LustreError: 234782:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.1@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7907.312642] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7907.317282] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7907.328889] LustreError: Skipped 1 previous similar message [ 7907.341788] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7907.345343] Lustre: Skipped 1 previous similar message [ 7907.371241] Lustre: Skipped 1 previous similar message [ 7918.055823] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7918.069621] Lustre: Skipped 3 previous similar messages [ 7919.073851] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7919.119385] LustreError: 236240:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7919.123064] LustreError: 236240:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7919.349975] Lustre: server umount lustre-OST0000 complete [ 7927.786571] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7927.795394] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7930.556833] Lustre: server umount lustre-MDT0000 complete [ 7933.410376] LustreError: 232543:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7933.454420] LustreError: 232543:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 13 previous similar messages [ 7935.036946] LustreError: 232523:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768562179 with bad export cookie 5686631367233906834 [ 7935.051937] LustreError: 232523:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7935.053726] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7941.324233] LustreError: 236647:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 7941.334542] LustreError: 236647:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 7941.678684] Lustre: server umount lustre-MDT0001 complete [ 7954.331836] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [ 7957.909879] Key type lgssc unregistered [ 7958.287746] LNet: 237180:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7958.293607] LNetError: 237180:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7958.315064] LNet: Removed LNI 192.168.203.101@tcp [ 7959.327661] Key type .llcrypt unregistered [ 7959.331865] Key type ._llcrypt unregistered [ 8005.114965] Key type ._llcrypt registered [ 8005.122412] Key type .llcrypt registered [ 8005.370124] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 8007.025533] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8007.062474] alg: No test for adler32 (adler32-zlib) [ 8008.557510] Lustre: Lustre: Build Version: 2.17.50_30_gc0571ea [ 8008.795849] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [ 8010.480231] Key type lgssc registered [ 8011.833881] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8019.792675] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8019.807859] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8021.136819] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8021.180896] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8021.562897] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8026.031729] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 8035.790963] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8035.986649] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8036.034913] Lustre: 239368:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8036.541078] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8041.322901] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 8047.695819] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8051.962068] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8059.371091] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8059.607558] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8059.856119] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8066.737501] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 8067.254735] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 8073.398284] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8128.481796] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8128.498900] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8128.521654] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8132.628106] LustreError: 241727:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 8132.887897] Lustre: server umount lustre-OST0000 complete [ 8137.710282] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 8151.351809] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8151.927559] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8158.825480] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 8165.141335] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8169.331224] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8169.356744] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8169.361564] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8210.406470] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8210.426527] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8210.429077] Lustre: Skipped 1 previous similar message [ 8210.440662] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8210.467602] Lustre: Skipped 2 previous similar messages [ 8212.610736] LustreError: 243390:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 8212.614500] LustreError: 243390:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8213.114735] Lustre: server umount lustre-OST0000 complete [ 8220.645936] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8220.654621] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8220.662768] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8220.668689] Lustre: Skipped 2 previous similar messages [ 8220.685300] Lustre: Skipped 1 previous similar message [ 8223.370028] LustreError: 243591:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 8223.391500] LustreError: 243591:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8223.579582] Lustre: server umount lustre-MDT0000 complete [ 8227.371608] LustreError: 239362:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768562471 with bad export cookie 4750346278052947733 [ 8227.372826] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8227.386867] LustreError: 239362:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8227.537945] LustreError: 243794:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8227.554603] LustreError: 243794:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8227.849351] Lustre: server umount lustre-MDT0001 complete [ 8242.007465] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [ 8244.960447] Key type lgssc unregistered [ 8245.314924] LNet: 244326:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8245.330173] LNetError: 244326:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8245.365055] LNet: Removed LNI 192.168.203.101@tcp [ 8246.389225] Key type .llcrypt unregistered [ 8246.391617] Key type ._llcrypt unregistered [ 8274.202987] Key type ._llcrypt registered [ 8274.205920] Key type .llcrypt registered [ 8274.322360] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 8275.217875] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8275.241358] alg: No test for adler32 (adler32-zlib) [ 8276.373892] Lustre: Lustre: Build Version: 2.17.50_30_gc0571ea [ 8276.658751] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [ 8278.433690] Key type lgssc registered [ 8279.631679] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8288.167924] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8288.191757] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8289.836916] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8293.948604] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 8298.987615] LustreError: 245724:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8298.999557] LustreError: 245724:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8302.542779] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8302.941801] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8307.545768] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 8314.756143] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8320.070536] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8327.109149] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8327.625668] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8334.839820] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 8334.884738] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 8340.300862] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8363.492294] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8363.503601] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8363.525507] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8365.543813] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8365.545197] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8365.562654] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8367.000125] LustreError: 248741:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8367.234949] Lustre: server umount lustre-OST0000 complete [ 8374.768729] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8374.782196] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8374.801363] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8374.813819] Lustre: Skipped 1 previous similar message [ 8377.361549] LustreError: 248944:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 8377.368994] LustreError: 248944:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8377.702231] Lustre: server umount lustre-MDT0000 complete [ 8380.905404] LustreError: 245720:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8380.925458] LustreError: 245720:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8381.281574] LustreError: 248945:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768562625 with bad export cookie 14043537295409501345 [ 8381.287741] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8381.302945] LustreError: 248945:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8381.465372] LustreError: 249146:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 8381.469551] LustreError: 249146:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8381.788788] Lustre: server umount lustre-MDT0001 complete [ 8390.660287] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [ 8392.940733] Key type lgssc unregistered [ 8393.221624] LNet: 249678:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8393.228067] LNetError: 249678:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8393.247788] LNet: Removed LNI 192.168.203.101@tcp [ 8394.139621] Key type .llcrypt unregistered [ 8394.143121] Key type ._llcrypt unregistered [ 8411.255940] Key type ._llcrypt registered [ 8411.257493] Key type .llcrypt registered [ 8411.324903] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 8412.180987] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8412.214990] alg: No test for adler32 (adler32-zlib) [ 8413.409833] Lustre: Lustre: Build Version: 2.17.50_30_gc0571ea [ 8413.634077] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [ 8415.360170] Key type lgssc registered [ 8416.877980] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8425.548283] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8425.567723] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8427.048093] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8431.568898] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 8437.222254] LustreError: 250880:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8437.251514] LustreError: 250880:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8440.372396] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8440.864560] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8445.801691] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 8451.907641] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8456.403286] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8462.641377] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8462.915726] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8469.615343] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 8472.143776] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8475.611294] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8487.396449] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8487.398774] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8487.411128] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8488.192091] LustreError: 253266:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8488.282194] Lustre: server umount lustre-OST0000 complete [ 8492.516462] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8492.520719] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8492.535170] Lustre: Skipped 2 previous similar messages [ 8492.563599] Lustre: Skipped 2 previous similar messages [ 8497.633547] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8497.639511] Lustre: Skipped 1 previous similar message [ 8498.444408] LustreError: 253467:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 8498.453802] LustreError: 253467:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8498.546573] Lustre: server umount lustre-MDT0000 complete [ 8502.455475] LustreError: 250861:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768562746 with bad export cookie 10351203227355138866 [ 8502.464068] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8502.473634] LustreError: 250861:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8502.633309] LustreError: 253668:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 8502.642650] LustreError: 253668:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8502.914216] Lustre: server umount lustre-MDT0001 complete [ 8514.130496] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [ 8516.922944] Key type lgssc unregistered [ 8517.298186] LNet: 254201:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8517.306854] LNetError: 254201:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8518.381547] LNet: Removed LNI 192.168.203.101@tcp [ 8519.341603] Key type .llcrypt unregistered [ 8519.348730] Key type ._llcrypt unregistered [ 8540.617725] Key type ._llcrypt registered [ 8540.620563] Key type .llcrypt registered [ 8540.697053] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 8541.869758] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8541.935756] alg: No test for adler32 (adler32-zlib) [ 8543.022936] Lustre: Lustre: Build Version: 2.17.50_30_gc0571ea [ 8543.206904] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [ 8544.856167] Key type lgssc registered [ 8546.068389] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8552.645862] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8552.660585] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8554.213349] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8558.216362] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 8564.707054] LustreError: 255578:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8564.720737] LustreError: 255578:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8566.263530] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8566.644824] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8571.310603] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 8576.813723] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8580.268662] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8586.259669] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8586.463551] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8592.384684] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 8596.859507] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8599.803144] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8622.562743] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8622.566145] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8622.578522] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8625.637307] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8625.640243] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8627.643819] LustreError: 258595:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8627.816448] Lustre: server umount lustre-OST0000 complete [ 8633.313107] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8633.322844] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8633.341038] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8633.345106] Lustre: Skipped 1 previous similar message [ 8635.873846] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8635.874997] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8637.274944] LustreError: 258797:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 8637.283499] LustreError: 258797:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8637.358709] Lustre: server umount lustre-MDT0000 complete [ 8640.052344] LustreError: 255560:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768562884 with bad export cookie 5583825479560058915 [ 8640.053038] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8640.059368] LustreError: 255560:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8640.200840] LustreError: 258999:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 8640.216140] LustreError: 258999:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8640.432715] Lustre: server umount lustre-MDT0001 complete [ 8649.796199] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [ 8652.574306] Key type lgssc unregistered [ 8652.901708] LNet: 259531:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8652.908485] LNetError: 259531:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8652.924553] LNet: Removed LNI 192.168.203.101@tcp [ 8653.524592] Key type .llcrypt unregistered [ 8653.526993] Key type ._llcrypt unregistered [ 8665.946432] Key type ._llcrypt registered [ 8665.948144] Key type .llcrypt registered [ 8666.041736] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 8667.064932] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8667.235483] alg: No test for adler32 (adler32-zlib) [ 8668.246973] Lustre: Lustre: Build Version: 2.17.50_30_gc0571ea [ 8668.464957] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [ 8670.144684] Key type lgssc registered [ 8670.990270] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8677.067547] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8677.079251] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8678.451505] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8681.840564] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 8688.391274] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8688.674842] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8692.358471] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 8697.167224] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8700.239300] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8706.863991] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8707.282798] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8713.668631] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 8719.717363] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8723.717515] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8734.178977] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8734.184860] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8734.197788] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8734.202492] Lustre: Skipped 1 previous similar message [ 8739.298379] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8739.305794] Lustre: Skipped 1 previous similar message [ 8739.577292] LustreError: 263139:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8739.638873] Lustre: server umount lustre-OST0000 complete [ 8744.418663] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8744.420928] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8744.439172] Lustre: Skipped 2 previous similar messages [ 8744.448388] Lustre: Skipped 1 previous similar message [ 8748.393711] LustreError: 263340:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 8748.397262] LustreError: 263340:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8748.567400] Lustre: server umount lustre-MDT0000 complete [ 8749.537640] LustreError: 260719:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8749.558429] LustreError: 260719:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 8751.644251] LustreError: 260693:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768562995 with bad export cookie 1453439356185666197 [ 8751.652558] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8751.786590] LustreError: 263542:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 8751.801156] LustreError: 263542:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8751.997675] Lustre: server umount lustre-MDT0001 complete [ 8761.231725] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [ 8763.236217] Key type lgssc unregistered [ 8763.462846] LNet: 264074:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8763.468055] LNetError: 264074:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8763.478994] LNet: Removed LNI 192.168.203.101@tcp [ 8764.048942] Key type .llcrypt unregistered [ 8764.051548] Key type ._llcrypt unregistered [ 8772.987700] Key type ._llcrypt registered [ 8772.989751] Key type .llcrypt registered [ 8773.067936] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8786.710869] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_hostid [ 8797.128545] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 8797.774357] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8797.901092] alg: No test for adler32 (adler32-zlib) [ 8798.888625] Lustre: Lustre: Build Version: 2.17.50_30_gc0571ea [ 8799.034284] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [ 8800.632188] Key type lgssc registered [ 8801.570154] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8829.559957] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8829.585417] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8830.861338] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8830.915265] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8831.042416] Lustre: lustre-MDT0000: new disk, initializing [ 8831.150084] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8831.172771] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8834.144798] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 8842.130828] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8842.199394] Lustre: 267550:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8842.234445] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8842.237209] Lustre: Skipped 1 previous similar message [ 8842.291971] Lustre: lustre-MDT0001: new disk, initializing [ 8842.350745] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8842.377566] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8842.386339] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8845.488231] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 8851.840365] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8854.764567] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8859.573672] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8859.796200] Lustre: lustre-OST0000: new disk, initializing [ 8859.800077] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8859.875877] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8864.714726] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 8865.331336] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8865.343129] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8865.466034] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8870.880150] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8874.318519] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8874.524181] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8878.509900] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8878.717795] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8881.127626] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8881.129527] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8881.136420] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8886.249498] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8886.265666] Lustre: Skipped 2 previous similar messages [ 8886.408848] LustreError: 270483:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 8886.606985] Lustre: server umount lustre-OST0000 complete [ 8891.363262] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8891.364568] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8891.369033] Lustre: Skipped 2 previous similar messages [ 8891.391908] Lustre: Skipped 1 previous similar message [ 8896.344270] LustreError: 270684:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 8896.349588] LustreError: 270684:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8896.406327] Lustre: server umount lustre-MDT0000 complete [ 8899.310072] LustreError: 267543:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768563143 with bad export cookie 1522806259493249381 [ 8899.314188] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8899.318709] LustreError: 267543:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8899.463855] LustreError: 270885:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8899.468690] LustreError: 270885:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8899.657466] Lustre: server umount lustre-MDT0001 complete [ 8906.707973] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 06:32:29 (1768563149) [ 8912.323472] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8912.639830] LustreError: 271555:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8912.652210] LustreError: 271555:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 8912.713409] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8916.523252] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 8917.986785] LustreError: 271555:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8922.090392] LustreError: 271555:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8923.703177] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8923.915805] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8927.949951] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 8932.470075] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8935.294593] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8940.407226] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8940.614590] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8945.033485] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 8949.325395] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8967.651423] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8967.653923] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8967.660607] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8967.664951] Lustre: Skipped 1 previous similar message [ 8967.675779] Lustre: Skipped 1 previous similar message [ 8972.288207] LustreError: 273819:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8972.296040] LustreError: 273819:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 8972.373542] Lustre: server umount lustre-OST0000 complete [ 8975.329774] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8975.336771] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8975.355392] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8981.196668] LustreError: 274021:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 8981.204625] LustreError: 274021:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8981.329610] Lustre: server umount lustre-MDT0000 complete [ 8983.011495] LustreError: 273539:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8983.032737] LustreError: 273539:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8984.598340] LustreError: 271536:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768563228 with bad export cookie 1522806259493250263 [ 8984.606200] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8984.624048] LustreError: 271536:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8984.853905] Lustre: server umount lustre-MDT0001 complete [ 8999.353992] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [ 9004.834263] Key type lgssc unregistered [ 9005.149685] LNet: 274756:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9005.154236] LNetError: 274756:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9005.176736] LNet: Removed LNI 192.168.203.101@tcp [ 9006.700474] Key type .llcrypt unregistered [ 9006.705150] Key type ._llcrypt unregistered [ 9042.475365] Key type ._llcrypt registered [ 9042.478608] Key type .llcrypt registered [ 9042.612555] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_hostid [ 9059.456719] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 9061.451885] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9061.509436] alg: No test for adler32 (adler32-zlib) [ 9062.638587] Lustre: Lustre: Build Version: 2.17.50_30_gc0571ea [ 9062.994031] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [ 9064.737340] Key type lgssc registered [ 9066.067929] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9118.207727] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9118.238332] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9119.627189] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9119.711106] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9119.907536] Lustre: lustre-MDT0000: new disk, initializing [ 9120.099230] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9120.118388] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9125.791836] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 9138.035257] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9138.247315] Lustre: 278549:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9138.344550] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9138.359242] Lustre: Skipped 1 previous similar message [ 9138.507190] Lustre: lustre-MDT0001: new disk, initializing [ 9138.637655] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9138.722789] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9138.737604] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9144.064658] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 9153.294628] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9158.431254] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9165.828699] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9166.137101] Lustre: lustre-OST2710: new disk, initializing [ 9166.141909] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 9166.251714] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 9173.797818] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 9174.661624] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 9174.687981] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 9174.900115] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 9182.864560] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 9191.294638] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9191.497333] Lustre: lustre-OST03e8: new disk, initializing [ 9191.501636] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 9191.593611] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 9198.569909] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 9200.234163] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 9200.249377] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 9200.440473] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 9205.863548] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 9213.616598] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 9213.825594] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9218.877841] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 9219.124823] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9223.842233] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 9224.163722] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9229.403690] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 9229.729109] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9235.515584] LustreError: 282886:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 9235.673609] Lustre: server umount lustre-MDT0000 complete [ 9235.937629] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9235.952737] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9236.462612] Lustre: lustre-MDT0000-lwp-OST2710: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9239.750377] LustreError: 278542:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768563483 with bad export cookie 11875549850449740000 [ 9239.755301] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9239.770384] LustreError: 278542:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9239.976609] LustreError: 283088:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 9239.990029] LustreError: 283088:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 9240.703664] Lustre: server umount lustre-MDT0001 complete [ 9255.692488] LustreError: 283289:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 9255.694732] LustreError: 283289:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 9255.744174] Lustre: server umount lustre-OST2710 complete [ 9257.637277] Lustre: 276347:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1768563485/real 1768563485] req@00000000330900e3 x1854473038268928/t0(0) o400->lustre-MDT0001-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1768563501 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9257.669109] Lustre: lustre-MDT0001-lwp-OST03e8: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9257.689976] Lustre: Skipped 2 previous similar messages [ 9259.867271] LustreError: 283491:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 9259.876275] LustreError: 283491:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 9260.151097] Lustre: server umount lustre-OST03e8 complete [ 9281.382064] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_hostid [ 9292.547572] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 9332.622906] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9333.047244] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9333.133930] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9333.305558] Lustre: lustre-MDT0000: new disk, initializing [ 9333.497653] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9333.527959] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9338.643521] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 9350.738513] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9350.858471] Lustre: 286281:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9350.903912] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9350.907423] Lustre: Skipped 1 previous similar message [ 9350.973684] Lustre: lustre-MDT0001: new disk, initializing [ 9351.122806] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9351.195442] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9351.213680] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9356.452897] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 9364.419610] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9368.489837] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9375.666113] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9375.885329] Lustre: lustre-OST0000: new disk, initializing [ 9375.888409] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9375.955730] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9381.000708] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 9383.549236] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9383.571621] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9383.718717] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9389.966705] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9395.685391] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9396.138782] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9401.152901] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9401.540880] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9404.396029] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9404.405056] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9404.428037] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9404.449126] Lustre: Skipped 1 previous similar message [ 9409.511403] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9409.524853] Lustre: Skipped 1 previous similar message [ 9414.625877] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9414.637153] Lustre: Skipped 1 previous similar message [ 9418.208142] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9418.279311] LustreError: 289215:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 9418.293406] LustreError: 289215:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 9418.379264] Lustre: server umount lustre-OST0000 complete [ 9422.817694] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9422.824182] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9422.835678] Lustre: Skipped 1 previous similar message [ 9422.839632] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9428.952932] LustreError: 289417:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 9428.971857] LustreError: 289417:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 9429.472596] Lustre: server umount lustre-MDT0000 complete [ 9429.993601] LustreError: 288079:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9430.012588] LustreError: 288079:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 9432.790557] LustreError: 286272:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768563676 with bad export cookie 11875549850449741687 [ 9432.793916] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9432.803022] LustreError: 286272:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9433.450508] Lustre: server umount lustre-MDT0001 complete [ 9444.075278] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 06:41:25 (1768563685) [ 9445.832861] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 9448.095433] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 06:41:29 (1768563689) [ 9452.685368] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 9485.729476] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9486.246572] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9486.285446] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9486.579347] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9491.852217] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 9501.159824] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9501.307681] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9501.365754] Lustre: 291716:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9505.816887] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 9512.768469] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9516.627100] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9535.667317] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9535.877602] Lustre: Denying initial registration attempt from nid 192.168.203.101@tcp, specified as failover [ 9535.884592] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9535.891465] LustreError: 293328:0:(tgt_mount.c:2483:server_fill_super()) Unable to start targets: -99 [ 9535.900972] LustreError: 293328:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [ 9535.913631] LustreError: 293328:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 9536.005684] Lustre: server umount lustre-OST0000 complete [ 9536.008610] LustreError: 293328:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9547.762735] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9547.771116] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9547.780525] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9547.780534] Lustre: Skipped 2 previous similar messages [ 9547.787808] Lustre: Skipped 2 previous similar messages [ 9549.686197] LustreError: 293696:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 9549.691802] LustreError: 293696:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 9550.202822] Lustre: server umount lustre-MDT0000 complete [ 9552.866568] LustreError: 291727:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9552.902582] LustreError: 291727:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9554.911294] LustreError: 291707:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768563799 with bad export cookie 11875549850449742576 [ 9554.934095] LustreError: 291707:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9554.937247] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9555.673381] Lustre: server umount lustre-MDT0001 complete [ 9569.090524] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [ 9572.436967] Key type lgssc unregistered [ 9572.705520] LNet: 294433:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9572.713801] LNetError: 294433:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9573.741011] LNet: Removed LNI 192.168.203.101@tcp [ 9574.784570] Key type .llcrypt unregistered [ 9574.792609] Key type ._llcrypt unregistered [ 9587.676103] Key type ._llcrypt registered [ 9587.677837] Key type .llcrypt registered [ 9587.855906] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 9589.093843] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9589.173839] alg: No test for adler32 (adler32-zlib) [ 9590.326705] Lustre: Lustre: Build Version: 2.17.50_30_gc0571ea [ 9590.645851] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [ 9592.384195] Key type lgssc registered [ 9593.714107] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9655.621424] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 9664.619176] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9664.644879] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9665.951639] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9665.984740] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9666.226293] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9671.296233] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 9680.088829] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9680.173387] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9680.208671] Lustre: 297171:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9680.512257] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9685.290599] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 9691.809595] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9696.295273] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9714.391317] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9714.617936] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9714.867134] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9721.238597] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 9727.540436] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9734.626358] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9734.632657] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9734.653109] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9734.657828] Lustre: Skipped 1 previous similar message [ 9739.748231] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9744.353729] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9744.412032] LustreError: 299450:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 9744.647542] Lustre: server umount lustre-OST0000 complete [ 9749.989883] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9750.005748] Lustre: Skipped 2 previous similar messages [ 9750.006727] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9750.024298] Lustre: Skipped 2 previous similar messages [ 9755.110302] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9755.116973] Lustre: Skipped 1 previous similar message [ 9760.229421] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9760.239036] Lustre: Skipped 1 previous similar message [ 9763.296386] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9763.421444] LustreError: 299654:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 9763.436746] LustreError: 299654:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 9763.554288] Lustre: server umount lustre-MDT0000 complete [ 9767.299094] LustreError: 298811:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768564011 with bad export cookie 4310563958837724040 [ 9767.301323] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9767.304663] LustreError: 298811:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9767.429254] LustreError: 299856:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 9767.443580] LustreError: 299856:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 9767.621516] Lustre: server umount lustre-MDT0001 complete [ 9778.452286] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [ 9781.151651] Key type lgssc unregistered [ 9781.563686] LNet: 300388:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9781.572547] LNetError: 300388:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9781.607747] LNet: Removed LNI 192.168.203.101@tcp [ 9782.420364] Key type .llcrypt unregistered [ 9782.422818] Key type ._llcrypt unregistered [ 9807.440320] Key type ._llcrypt registered [ 9807.443626] Key type .llcrypt registered [ 9807.544226] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 9808.825438] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9808.908679] alg: No test for adler32 (adler32-zlib) [ 9810.054393] Lustre: Lustre: Build Version: 2.17.50_30_gc0571ea [ 9810.362911] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [ 9812.064152] Key type lgssc registered [ 9813.537185] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9822.015227] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9822.065110] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9823.535840] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9827.431343] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 9832.933049] LustreError: 301789:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9832.947290] LustreError: 301789:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9836.219320] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9836.704414] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9841.642264] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 9846.877361] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9850.652271] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9857.071125] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9857.341378] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9863.043819] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 9868.253422] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9877.481760] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9877.488593] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9877.500872] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9879.015733] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9879.018290] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9879.036075] Lustre: Skipped 1 previous similar message [ 9881.346554] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.1@tcp (stopping) [ 9883.333312] LustreError: 304033:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 9883.613345] Lustre: server umount lustre-MDT0000 complete [ 9884.136117] LustreError: 302760:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9884.166927] LustreError: 302760:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9886.432623] LustreError: 303893:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.1@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9887.575313] LustreError: 301771:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768564131 with bad export cookie 1149094092025289154 [ 9887.577000] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9887.589700] LustreError: 301771:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9889.253210] LustreError: 303893:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9889.255097] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9889.297501] Lustre: Skipped 1 previous similar message [ 9889.308300] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9893.813862] LustreError: 304234:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 9893.830787] LustreError: 304234:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 9894.080629] Lustre: server umount lustre-MDT0001 complete [ 9900.027931] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9908.321468] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9913.169279] LustreError: 303394:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9913.187336] LustreError: 303394:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9913.254509] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9913.309673] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:257) [ 9917.513271] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 9922.473440] LustreError: 303393:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9922.481560] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9922.486741] LustreError: 303393:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9926.229675] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9926.715887] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9931.384600] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 9931.757051] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9937.078603] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9938.523263] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9941.934445] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9943.449508] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9949.171886] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9949.192389] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9949.208607] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9949.223493] Lustre: Skipped 2 previous similar messages [ 9952.234694] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9954.311459] LustreError: 306536:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 9954.325667] LustreError: 306536:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 9954.520516] Lustre: server umount lustre-OST0000 complete [ 9962.465616] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9962.466754] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9962.468600] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9962.468608] Lustre: Skipped 2 previous similar messages [ 9962.476064] Lustre: Skipped 2 previous similar messages [ 9964.407165] LustreError: 306738:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 9964.411195] LustreError: 306738:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 9964.738738] Lustre: server umount lustre-MDT0000 complete [ 9967.588642] LustreError: 305047:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9967.610334] LustreError: 305047:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9968.680875] LustreError: 301771:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768564212 with bad export cookie 1149094092025296756 [ 9968.688981] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9968.692041] LustreError: 301771:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9968.853381] LustreError: 306941:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 9968.861540] LustreError: 306941:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 9969.129353] Lustre: server umount lustre-MDT0001 complete [ 9980.436832] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [ 9983.484523] Key type lgssc unregistered [ 9983.814285] LNet: 307473:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9983.822984] LNetError: 307473:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9983.849947] LNet: Removed LNI 192.168.203.101@tcp [ 9984.795656] Key type .llcrypt unregistered [ 9984.798152] Key type ._llcrypt unregistered [10009.102658] Key type ._llcrypt registered [10009.107506] Key type .llcrypt registered [10009.299160] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [10010.661025] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10010.715781] alg: No test for adler32 (adler32-zlib) [10011.815796] Lustre: Lustre: Build Version: 2.17.50_30_gc0571ea [10012.061885] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [10013.776213] Key type lgssc registered [10015.077915] Lustre: Echo OBD driver; http://www.lustre.org/ [10023.600247] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10023.617235] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10025.322655] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10029.086484] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [10035.695254] LustreError: 308871:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10035.709679] LustreError: 308871:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10036.458815] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10036.714489] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10040.433378] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [10045.797686] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10049.427836] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10055.140037] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10055.151228] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10058.434873] LustreError: 310259:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [10058.610646] Lustre: server umount lustre-MDT0000 complete [10060.257871] LustreError: 308870:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10060.299073] LustreError: 308870:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10061.284775] LustreError: 309894:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10061.297811] LustreError: 309894:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10061.759624] LustreError: 308850:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768564305 with bad export cookie 1642750598295569919 [10061.764749] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10061.777172] LustreError: 308850:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [10061.941773] LustreError: 310461:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [10061.947589] LustreError: 310461:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [10062.135726] Lustre: server umount lustre-MDT0001 complete [10076.797651] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10077.060435] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10077.095782] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10077.315926] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10081.537586] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [10089.431948] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10089.579605] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10089.613786] Lustre: 311233:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10089.855696] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10093.937871] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [10099.911454] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10103.410741] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10109.985965] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10110.126090] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [10110.137618] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [10110.146323] LustreError: 312886:0:(tgt_mount.c:2483:server_fill_super()) Unable to start targets: -2 [10110.154819] LustreError: 312886:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [10110.164598] LustreError: 312886:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [10110.248538] Lustre: server umount lustre-OST0000 complete [10110.251688] LustreError: 312886:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -2 [10117.314478] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10117.466808] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10117.634598] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10122.951480] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [10123.384699] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:289) [10127.519159] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10133.712672] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10133.789257] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [10133.842625] Lustre: lustre-OST0001: new disk, initializing [10133.845377] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10133.913561] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10139.006454] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [10140.773913] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10140.784417] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10140.919884] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10143.870399] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [10151.393655] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10151.397156] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10151.404565] LustreError: Skipped 1 previous similar message [10151.410336] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10151.422545] Lustre: Skipped 2 previous similar messages [10151.445145] Lustre: Skipped 2 previous similar messages [10152.765130] LustreError: 315003:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [10152.771686] LustreError: 315003:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [10152.815535] Lustre: server umount lustre-OST0001 complete [10156.516480] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10156.517453] LustreError: 314764:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10156.541799] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10156.550501] LustreError: 314764:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10156.561179] Lustre: Skipped 1 previous similar message [10161.639636] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10161.647419] Lustre: Skipped 1 previous similar message [10161.647601] LustreError: 313353:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10161.661213] LustreError: 313353:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10161.715138] LustreError: 315205:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [10161.718234] LustreError: 315205:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [10161.814716] Lustre: server umount lustre-OST0000 complete [10166.754660] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10166.754972] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10166.762067] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10166.781674] Lustre: Skipped 2 previous similar messages [10171.062798] LustreError: 315406:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [10171.066315] LustreError: 315406:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [10171.186434] Lustre: server umount lustre-MDT0000 complete [10171.877205] LustreError: 311244:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10171.889078] LustreError: 311244:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10174.408823] LustreError: 311224:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768564418 with bad export cookie 1642750598295570472 [10174.416656] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10174.426277] LustreError: 311224:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [10174.684545] Lustre: server umount lustre-MDT0001 complete [10183.692635] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [10185.730735] Key type lgssc unregistered [10185.983517] LNet: 316138:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10185.989681] LNetError: 316138:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [10186.002340] LNet: Removed LNI 192.168.203.101@tcp [10186.636353] Key type .llcrypt unregistered [10186.638112] Key type ._llcrypt unregistered [10248.378708] Key type ._llcrypt registered [10248.382199] Key type .llcrypt registered [10248.470976] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_hostid [10259.200553] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [10259.937308] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10260.011053] alg: No test for adler32 (adler32-zlib) [10260.996778] Lustre: Lustre: Build Version: 2.17.50_30_gc0571ea [10261.191518] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [10262.816162] Key type lgssc registered [10263.819178] Lustre: Echo OBD driver; http://www.lustre.org/ [10291.932946] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10291.957212] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10293.223939] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10293.250614] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10293.305902] Lustre: lustre-MDT0000: new disk, initializing [10293.380355] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10293.399311] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10296.941228] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [10306.362336] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10306.441926] Lustre: 321016:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10306.477853] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10306.481942] Lustre: Skipped 1 previous similar message [10306.572092] Lustre: lustre-MDT0001: new disk, initializing [10306.654557] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10306.687543] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10306.699189] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10309.908578] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [10316.538579] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10319.411831] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10325.109950] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10325.302155] Lustre: lustre-OST0000: new disk, initializing [10325.305461] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10325.383264] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10330.142611] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [10334.788864] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10334.806931] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10334.886325] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10336.682526] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10339.952716] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10340.108324] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10343.484471] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10343.614024] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10345.441972] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10345.446162] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10345.452308] Lustre: Skipped 1 previous similar message [10345.454915] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10345.466168] Lustre: Skipped 1 previous similar message [10350.563244] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10355.691870] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10355.699611] Lustre: Skipped 2 previous similar messages [10359.264138] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10359.313351] LustreError: 323944:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [10359.433107] Lustre: server umount lustre-OST0000 complete [10362.849895] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10362.859136] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10362.874618] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10365.924868] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10368.436808] LustreError: 324147:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [10368.440176] LustreError: 324147:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [10368.686389] Lustre: server umount lustre-MDT0000 complete [10371.635540] LustreError: 321007:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768564615 with bad export cookie 13854443104163184846 [10371.649453] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10371.653603] LustreError: 321007:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [10371.790563] LustreError: 324349:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [10371.805668] LustreError: 324349:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [10371.960117] Lustre: server umount lustre-MDT0001 complete [10379.022403] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 06:57:01 (1768564621) [10384.773299] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 06:57:07 (1768564627) [10390.921530] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10391.175294] LustreError: 325364:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10391.200576] LustreError: 325364:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10391.253596] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10394.305428] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [10396.641474] LustreError: 325364:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10400.507789] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10400.817202] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10403.903498] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [10407.970861] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10410.621802] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10415.437857] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10415.681779] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10419.612980] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [10424.021480] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10431.021500] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [10432.184618] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [10433.332940] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [10434.488908] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [10435.905737] Lustre: Failing over lustre-MDT0000 [10436.037372] LustreError: 327795:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [10436.040095] LustreError: 327795:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [10436.108403] Lustre: server umount lustre-MDT0000 complete [10436.576961] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10436.581841] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10436.589995] LustreError: 326073:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10440.674342] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10440.678051] LustreError: 326073:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10440.707499] LustreError: 326073:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10445.792997] LustreError: 326073:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10445.802017] LustreError: 326073:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10452.320282] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10452.455428] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10452.741664] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10452.790198] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10453.246143] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10455.821253] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [10458.110459] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10458.153055] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [10458.226779] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10461.968322] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10463.277836] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10466.538223] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [10468.146291] Lustre: Failing over lustre-MDT0000 [10468.301472] LustreError: 328871:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [10468.308618] LustreError: 328871:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [10468.331679] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10468.334143] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10468.339167] Lustre: Skipped 2 previous similar messages [10468.339455] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10468.356917] Lustre: Skipped 4 previous similar messages [10468.444405] Lustre: server umount lustre-MDT0000 complete [10468.590265] LustreError: 325360:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.1@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10468.624869] LustreError: 325360:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [10484.756908] LustreError: 325359:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.1@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10484.795834] LustreError: 325359:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 12 previous similar messages [10486.162702] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10486.284240] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10486.514520] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10486.561582] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10490.080106] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10490.242610] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [10491.892214] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10491.902257] Lustre: Skipped 2 previous similar messages [10491.924212] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [10491.988299] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [10497.337804] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10498.575306] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10507.234629] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10507.237070] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10507.247159] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10507.256539] Lustre: Skipped 2 previous similar messages [10509.132554] LustreError: 329903:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [10509.137635] LustreError: 329903:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [10509.207023] Lustre: server umount lustre-MDT0000 complete [10512.018901] LustreError: 325343:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768564756 with bad export cookie 13854443104163187541 [10512.023829] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10512.041587] LustreError: 325343:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [10512.279842] Lustre: server umount lustre-MDT0001 complete [10525.906457] LustreError: 330305:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [10525.912261] LustreError: 330305:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [10526.078170] Lustre: server umount lustre-OST0000 complete [10539.998085] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10540.463444] LustreError: 331133:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10544.392700] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [10551.069730] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10551.417874] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10551.425568] Lustre: Skipped 1 previous similar message [10555.445466] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [10560.362322] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10563.672200] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10569.723328] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10575.763616] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [10581.299849] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10585.404504] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [10591.713901] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10591.722260] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10591.744243] Lustre: Skipped 1 previous similar message [10591.751797] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10591.754840] Lustre: Skipped 2 previous similar messages [10597.056279] LustreError: 333372:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [10597.059435] LustreError: 333372:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [10597.191457] Lustre: server umount lustre-OST0000 complete [10606.643391] Lustre: server umount lustre-MDT0000 complete [10609.780217] LustreError: 331114:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768564853 with bad export cookie 13854443104163188248 [10609.782512] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10609.789691] LustreError: 331114:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [10610.053522] Lustre: server umount lustre-MDT0001 complete [10619.722415] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [10622.116631] Key type lgssc unregistered [10622.397154] LNet: 334321:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10622.400638] LNetError: 334321:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [10622.414855] LNet: Removed LNI 192.168.203.101@tcp [10623.067453] Key type .llcrypt unregistered [10623.069475] Key type ._llcrypt unregistered [10642.865810] Key type ._llcrypt registered [10642.868342] Key type .llcrypt registered [10642.938438] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [10643.596168] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10643.698548] alg: No test for adler32 (adler32-zlib) [10644.659562] Lustre: Lustre: Build Version: 2.17.50_30_gc0571ea [10644.793419] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [10646.416265] Key type lgssc registered [10647.165967] Lustre: Echo OBD driver; http://www.lustre.org/ [10652.962960] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10652.998183] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10654.384566] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10657.494717] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [10663.579546] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10663.835615] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10666.991652] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [10671.280868] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10673.560757] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10677.860341] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10678.088970] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10679.145899] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [10682.313427] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [10686.292384] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10698.724294] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10698.731870] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10698.742468] Lustre: Skipped 1 previous similar message [10698.761328] Lustre: Skipped 2 previous similar messages [10701.340159] LustreError: 338037:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [10701.474857] Lustre: server umount lustre-MDT0000 complete [10703.847594] LustreError: 335794:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10703.860592] LustreError: 335794:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [10704.864810] LustreError: 335778:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768564948 with bad export cookie 4560766905958323086 [10704.874723] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10704.963479] LustreError: 338238:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [10704.967265] LustreError: 338238:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [10705.093996] Lustre: server umount lustre-MDT0001 complete [10718.409572] LustreError: 338439:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [10718.413829] LustreError: 338439:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [10718.516698] Lustre: server umount lustre-OST0000 complete [10728.702126] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10729.111075] LustreError: 339169:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10729.155447] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10732.872923] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [10734.561536] LustreError: 339169:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10739.514944] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10739.739529] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10743.765409] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [10748.250733] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10751.330244] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10756.996598] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10757.299229] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10762.403130] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [10766.814162] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10774.704167] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [10790.369820] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10790.376731] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10790.382398] Lustre: Skipped 3 previous similar messages [10790.393199] Lustre: Skipped 2 previous similar messages [10793.829597] LustreError: 341536:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [10793.836983] LustreError: 341536:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [10793.899635] Lustre: server umount lustre-MDT0000 complete [10795.491802] LustreError: 341222:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10795.510554] LustreError: 341222:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10796.339390] LustreError: 339149:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768565040 with bad export cookie 4560766905958324178 [10796.346882] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10796.347937] LustreError: 339149:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 8 previous similar messages [10796.592914] Lustre: server umount lustre-MDT0001 complete [10809.592710] LustreError: 341940:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [10809.595933] LustreError: 341940:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [10809.660517] Lustre: server umount lustre-OST0000 complete [10821.625706] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10821.878043] LustreError: 342769:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10821.919788] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10824.748488] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [10830.492561] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10833.798290] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [10838.304853] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10841.463863] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10846.528161] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10847.736284] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [10850.703259] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [10854.430806] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10857.957173] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10857.964140] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10857.975373] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10861.030899] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10861.036475] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10861.037196] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10863.511212] LustreError: 345008:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [10863.517099] LustreError: 345008:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [10863.617572] Lustre: server umount lustre-OST0000 complete [10866.662699] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10866.666117] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10866.685200] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10866.693748] Lustre: Skipped 1 previous similar message [10872.750346] Lustre: server umount lustre-MDT0000 complete [10875.318224] LustreError: 343487:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768565119 with bad export cookie 4560766905958325207 [10875.329980] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10875.524775] Lustre: server umount lustre-MDT0001 complete [10883.526420] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [10885.404253] Key type lgssc unregistered [10885.605656] LNet: 345941:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10885.609515] LNetError: 345941:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [10885.623191] LNet: Removed LNI 192.168.203.101@tcp [10886.216476] Key type .llcrypt unregistered [10886.218475] Key type ._llcrypt unregistered [10902.575402] Key type ._llcrypt registered [10902.576927] Key type .llcrypt registered [10902.650124] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [10903.571896] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10903.589658] alg: No test for adler32 (adler32-zlib) [10904.555535] Lustre: Lustre: Build Version: 2.17.50_30_gc0571ea [10904.712384] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [10906.328160] Key type lgssc registered [10907.113239] Lustre: Echo OBD driver; http://www.lustre.org/ [10912.802524] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10912.821429] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10912.827457] LustreError: 347349:0:(osd_handler.c:8561:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [10912.843422] LustreError: 347349:0:(obd_config.c:783:class_setup()) setup lustre-MDT0000-osd failed (-22) [10912.851504] LustreError: 347349:0:(obd_mount.c:193:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [10912.860193] LustreError: 347349:0:(tgt_mount.c:2451:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [10912.867869] LustreError: 347349:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -22 [10918.217367] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: user_xattr,acl,no_mbcache,nodelalloc [10918.220806] LustreError: 347746:0:(osd_handler.c:8561:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [10918.234516] LustreError: 347746:0:(obd_config.c:783:class_setup()) setup lustre-OST0000-osd failed (-22) [10918.238459] LustreError: 347746:0:(obd_mount.c:193:lustre_start_simple()) lustre-OST0000-osd setup error -22 [10918.243262] LustreError: 347746:0:(tgt_mount.c:2451:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [10918.251710] LustreError: 347746:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -22 [10930.789478] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [10932.605639] Key type lgssc unregistered [10932.791741] LNet: 348731:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10932.796656] LNetError: 348731:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [10932.809234] LNet: Removed LNI 192.168.203.101@tcp [10933.288963] Key type .llcrypt unregistered [10933.291483] Key type ._llcrypt unregistered [10945.658872] Key type ._llcrypt registered [10945.663160] Key type .llcrypt registered [10945.739822] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_hostid [10956.099417] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [10956.633115] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10956.648415] alg: No test for adler32 (adler32-zlib) [10957.633520] Lustre: Lustre: Build Version: 2.17.50_30_gc0571ea [10957.792850] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [10959.424118] Key type lgssc registered [10960.174777] Lustre: Echo OBD driver; http://www.lustre.org/ [10982.030919] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10982.076061] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10983.203167] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10983.225324] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10983.276271] Lustre: lustre-MDT0000: new disk, initializing [10983.315861] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10983.325618] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10986.018245] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [10992.949243] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10993.035217] Lustre: 351844:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10993.071434] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10993.073572] Lustre: Skipped 1 previous similar message [10993.119434] Lustre: lustre-MDT0001: new disk, initializing [10993.167615] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10993.188589] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10993.196672] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10996.188529] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [11001.205563] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11003.577544] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11007.482448] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11007.628584] Lustre: lustre-OST0000: new disk, initializing [11007.631576] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11007.686903] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11010.913828] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [11013.659068] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11013.674500] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11013.794174] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11016.183724] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11019.544914] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11019.672464] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11022.790138] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11022.874927] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11024.354634] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11024.354921] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11024.363754] LustreError: Skipped 1 previous similar message [11024.364597] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11024.390467] Lustre: Skipped 1 previous similar message [11028.962544] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11028.968486] Lustre: Skipped 2 previous similar messages [11030.259134] LustreError: 354774:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [11030.333211] Lustre: server umount lustre-OST0000 complete [11034.081830] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11034.086890] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11034.089070] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11034.093618] Lustre: Skipped 1 previous similar message [11039.205461] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11039.209634] Lustre: Skipped 2 previous similar messages [11044.325149] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11044.332149] Lustre: Skipped 1 previous similar message [11046.880115] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11046.966610] LustreError: 354975:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [11046.977892] LustreError: 354975:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [11047.048222] Lustre: server umount lustre-MDT0000 complete [11049.320666] LustreError: 351836:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768565293 with bad export cookie 13489437417734427408 [11049.324064] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11049.329726] LustreError: 351836:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [11049.475957] LustreError: 355177:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [11049.479142] LustreError: 355177:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [11049.602531] Lustre: server umount lustre-MDT0001 complete [11054.643537] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [11055.921714] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 07:08:18 (1768565298) [11059.835027] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11060.175529] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11062.380644] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [11065.314169] LustreError: 355896:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11065.320475] LustreError: 355896:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11066.720636] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11066.924973] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11068.831476] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [11071.915744] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11074.028487] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11078.078798] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11078.313513] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11081.906801] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [11085.115929] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11089.288879] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11089.384436] Lustre: lustre-OST0001: new disk, initializing [11089.386630] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11093.095778] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [11097.765684] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [11099.189769] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11099.197335] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11099.249064] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11104.229670] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11104.235221] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11104.247532] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11106.292133] LustreError: 359089:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [11106.297881] LustreError: 359089:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [11106.366647] Lustre: server umount lustre-OST0001 complete [11109.860778] LustreError: 357487:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11109.872626] LustreError: 357487:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11114.982994] LustreError: 357490:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11114.994750] LustreError: 357490:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11120.098404] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11120.099782] LustreError: 357487:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11120.103784] Lustre: Skipped 2 previous similar messages [11120.110246] LustreError: 357487:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11125.218500] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11125.220648] Lustre: Skipped 5 previous similar messages [11125.222112] LustreError: 358998:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11125.227898] LustreError: 358998:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11126.134124] LustreError: 359311:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [11126.136474] LustreError: 359311:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [11126.179909] Lustre: server umount lustre-OST0000 complete [11128.292684] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11128.300508] LustreError: Skipped 1 previous similar message [11128.302911] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11134.085375] Lustre: server umount lustre-MDT0000 complete [11135.457915] LustreError: 355896:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11135.473680] LustreError: 355896:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11135.912414] LustreError: 355876:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768565380 with bad export cookie 13489437417734428290 [11135.918578] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11135.919650] LustreError: 355876:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [11150.304293] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11150.355180] LustreError: 359716:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [11150.359836] LustreError: 359716:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [11150.503023] Lustre: server umount lustre-MDT0001 complete [11157.641493] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [11159.659599] Key type lgssc unregistered [11159.858386] LNet: 360251:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11159.868549] LNetError: 360251:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [11159.884956] LNet: Removed LNI 192.168.203.101@tcp [11160.382722] Key type .llcrypt unregistered [11160.384740] Key type ._llcrypt unregistered [11188.117086] Key type ._llcrypt registered [11188.119028] Key type .llcrypt registered [11188.190306] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [11188.762281] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11188.792158] alg: No test for adler32 (adler32-zlib) [11189.680316] Lustre: Lustre: Build Version: 2.17.50_30_gc0571ea [11189.790329] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [11191.400238] Key type lgssc registered [11192.022568] Lustre: Echo OBD driver; http://www.lustre.org/ [11195.716091] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11195.733335] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11196.884723] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [11196.898434] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11196.995472] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11199.234318] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [11203.982745] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11204.047820] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11204.073262] Lustre: 363008:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11204.235430] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11206.047115] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [11208.653131] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11210.318972] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11214.821431] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11214.827108] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11214.839946] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11216.355379] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11216.355504] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11216.364901] Lustre: Skipped 1 previous similar message [11217.423105] LustreError: 364371:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [11217.486618] Lustre: server umount lustre-MDT0000 complete [11219.622564] LustreError: 363001:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768565463 with bad export cookie 2529858225628608237 [11219.627182] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11219.628462] LustreError: 363001:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [11219.739081] LustreError: 364574:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [11219.746426] LustreError: 364574:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [11219.873814] Lustre: server umount lustre-MDT0001 complete [11224.954914] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 07:11:07 (1768565467) [11228.355255] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11228.645375] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11230.883620] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [11233.761046] LustreError: 365290:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11233.773013] LustreError: 365290:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11235.231118] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11235.445543] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11237.807851] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [11241.173883] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11243.561686] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11247.606161] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11247.713055] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [11247.829859] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11250.797458] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [11253.837321] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11264.013867] Lustre: Permanently deactivating lustre-OST0000 [11264.023512] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [11269.633451] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11270.751482] LustreError: 367933:0:(mgs_llog.c:1942:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [11270.759277] LustreError: 367933:0:(mgs_llog.c:1997:mgs_replace_nids()) Only MGS is allowed to be started [11270.762614] LustreError: 367933:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11272.008392] LustreError: 367981:0:(mgs_llog.c:1997:mgs_replace_nids()) Only MGS is allowed to be started [11272.012187] LustreError: 367981:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11278.969533] LustreError: 368079:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [11278.971616] LustreError: 368079:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [11279.022605] Lustre: server umount lustre-OST0000 complete [11280.373927] LustreError: 368231:0:(mgs_llog.c:1997:mgs_replace_nids()) Only MGS is allowed to be started [11280.378274] LustreError: 368231:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11281.890988] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11281.894825] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11281.902529] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11284.960828] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11284.961714] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11290.082270] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11290.084789] Lustre: Skipped 2 previous similar messages [11291.113534] LustreError: 368328:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [11291.122671] LustreError: 368328:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [11291.158669] Lustre: server umount lustre-MDT0000 complete [11293.227734] LustreError: 365270:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768565537 with bad export cookie 2529858225628608804 [11293.231465] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11293.233566] LustreError: 365270:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [11303.407172] LustreError: 368529:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [11303.416101] LustreError: 368529:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [11303.508831] Lustre: server umount lustre-MDT0001 complete [11309.310984] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11311.381962] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [11312.954051] LustreError: 369371:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [11316.156206] Lustre: 369710:0:(mgs_llog.c:1709:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.101@tcp [11317.212483] LustreError: 369807:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [11317.215593] LustreError: 369807:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [11317.270223] Lustre: server umount MGS complete [11323.135943] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11323.375751] LustreError: 370435:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11323.386828] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11323.390538] Lustre: Skipped 1 previous similar message [11323.419451] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11325.379490] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [11328.482403] LustreError: 370435:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11329.895816] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11330.061906] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11332.166273] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [11335.102235] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11337.132374] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11338.214624] Lustre: Permanently reactivating lustre-OST0000 [11338.222860] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [11338.227308] Lustre: Skipped 2 previous similar messages [11358.688202] LustreError: 371809:0:(osp_dev.c:692:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [11362.538932] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11362.739342] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11362.743778] Lustre: Skipped 1 previous similar message [11365.820270] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [11368.476195] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11378.146212] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11378.148644] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11378.149526] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11378.149691] LustreError: Skipped 1 previous similar message [11378.155883] Lustre: Skipped 1 previous similar message [11381.424070] LustreError: 373500:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [11381.429357] LustreError: 373500:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [11381.493681] Lustre: server umount lustre-OST0000 complete [11389.422808] Lustre: server umount lustre-MDT0000 complete [11391.298298] LustreError: 370415:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768565635 with bad export cookie 2529858225628609854 [11391.301650] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11391.307154] LustreError: 370415:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [11391.530819] Lustre: server umount lustre-MDT0001 complete [11396.877941] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [11398.318606] Key type lgssc unregistered [11398.482430] LNet: 374438:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11398.485786] LNetError: 374438:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [11398.496695] LNet: Removed LNI 192.168.203.101@tcp [11398.894348] Key type .llcrypt unregistered [11398.895765] Key type ._llcrypt unregistered [11407.224077] Key type ._llcrypt registered [11407.225379] Key type .llcrypt registered [11407.282884] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_hostid [11415.648882] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [11416.163347] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11416.221055] alg: No test for adler32 (adler32-zlib) [11417.137371] Lustre: Lustre: Build Version: 2.17.50_30_gc0571ea [11417.267835] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [11418.896232] Key type lgssc registered [11419.464657] Lustre: Echo OBD driver; http://www.lustre.org/ [11435.409520] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11435.422366] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11436.558971] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11436.584732] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11436.627734] Lustre: lustre-MDT0000: new disk, initializing [11436.668848] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11436.677809] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11438.577258] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [11443.730777] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11443.791114] Lustre: 377537:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11443.819526] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11443.821979] Lustre: Skipped 1 previous similar message [11443.861227] Lustre: lustre-MDT0001: new disk, initializing [11443.902401] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11443.923278] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11443.927193] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11445.796891] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [11449.816809] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11451.620492] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11454.846989] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11454.976783] Lustre: lustre-OST0000: new disk, initializing [11454.979136] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11455.022188] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11457.701265] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [11461.290257] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11461.635480] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11461.639702] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11461.690254] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11462.905706] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11463.034112] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11464.617723] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11464.704310] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11466.725381] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11466.726207] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11466.735254] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11466.740928] Lustre: Skipped 1 previous similar message [11467.505041] LustreError: 380468:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [11467.548636] Lustre: server umount lustre-OST0000 complete [11469.794184] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11469.797356] LustreError: Skipped 1 previous similar message [11469.799595] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11469.806406] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11469.808687] Lustre: Skipped 1 previous similar message [11471.841717] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11471.843159] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11475.341096] LustreError: 380669:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [11475.344157] LustreError: 380669:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [11475.380321] Lustre: server umount lustre-MDT0000 complete [11477.156413] LustreError: 379277:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768565721 with bad export cookie 10002968513026857770 [11477.159096] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11477.164845] LustreError: 379277:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [11477.260478] LustreError: 380871:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [11477.263131] LustreError: 380871:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [11477.375747] Lustre: server umount lustre-MDT0001 complete [11481.437132] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [11482.220136] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 07:15:25 (1768565725) [11485.599565] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11485.805129] LustreError: 381588:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11485.812655] LustreError: 381588:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [11485.833234] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11487.542293] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [11491.250612] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11491.304682] LustreError: 381588:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11491.435523] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11493.044900] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [11495.533394] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11497.433909] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11503.088277] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11503.660119] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11510.248496] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [11515.310950] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11517.354254] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [11520.936550] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [11531.233261] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11531.238523] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11531.259710] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11531.270628] Lustre: Skipped 1 previous similar message [11533.619154] LustreError: 384018:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [11533.621923] LustreError: 384018:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [11533.693992] Lustre: server umount lustre-OST0000 complete [11537.377337] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11537.446623] LustreError: Skipped 1 previous similar message [11537.470778] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11537.488780] Lustre: Skipped 1 previous similar message [11537.494608] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11537.503284] Lustre: Skipped 1 previous similar message [11547.105487] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11547.110268] Lustre: Skipped 3 previous similar messages [11551.200121] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11551.322616] LustreError: 384219:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [11551.338486] LustreError: 384219:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [11551.475969] Lustre: server umount lustre-MDT0000 complete [11555.197537] LustreError: 381569:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768565799 with bad export cookie 10002968513026858638 [11555.203234] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11555.776026] Lustre: server umount lustre-MDT0001 complete [11565.508253] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [11567.372818] Key type lgssc unregistered [11567.596587] LNet: 384954:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11567.603481] LNetError: 384954:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [11567.616042] LNet: Removed LNI 192.168.203.101@tcp [11568.186944] Key type .llcrypt unregistered [11568.193144] Key type ._llcrypt unregistered [11587.267650] Key type ._llcrypt registered [11587.269739] Key type .llcrypt registered [11587.373253] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [11587.577522] Key type .llcrypt unregistered [11587.585789] Key type ._llcrypt unregistered [11599.904459] Key type ._llcrypt registered [11599.906701] Key type .llcrypt registered [11599.971979] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [11600.774185] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11600.884191] alg: No test for adler32 (adler32-zlib) [11601.954586] Lustre: Lustre: Build Version: 2.17.50_30_gc0571ea [11602.177818] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [11603.840280] Key type lgssc registered [11604.601806] Lustre: Echo OBD driver; http://www.lustre.org/ [11610.384349] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11610.405441] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11611.766083] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11615.251966] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [11621.350507] LustreError: 387105:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11621.364230] LustreError: 387105:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11621.822301] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11622.057387] LustreError: 387105:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11622.079466] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11622.080303] LustreError: 387105:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11626.668419] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [11627.492415] LustreError: 387105:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11627.510944] LustreError: 387105:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11630.901899] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11632.617036] LustreError: 387105:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11632.633445] LustreError: 387105:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11636.413849] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11636.709599] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [11636.717263] Lustre: Skipped 1 previous similar message [11636.736465] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [11636.882080] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11640.310761] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [11652.065725] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11652.068633] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11652.069963] LustreError: Skipped 1 previous similar message [11652.070785] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11652.083485] Lustre: Skipped 1 previous similar message [11652.125214] Lustre: Skipped 1 previous similar message [11656.373132] LustreError: 389155:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [11656.545944] Lustre: server umount lustre-OST0000 complete [11662.309503] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11662.311763] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11662.318273] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11665.387347] LustreError: 389356:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [11665.392632] LustreError: 389356:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [11665.451900] Lustre: server umount lustre-MDT0000 complete [11667.886152] LustreError: 387085:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768565912 with bad export cookie 15202152588948313353 [11667.897140] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11667.901811] LustreError: 387085:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [11667.998389] LustreError: 389558:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [11668.001741] LustreError: 389558:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [11668.112636] Lustre: server umount lustre-MDT0001 complete [11675.422123] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [11677.343180] Key type lgssc unregistered [11677.507574] LNet: 390090:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11677.512340] LNetError: 390090:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [11677.522605] LNet: Removed LNI 192.168.203.101@tcp [11678.038676] Key type .llcrypt unregistered [11678.042478] Key type ._llcrypt unregistered [11690.185681] Key type ._llcrypt registered [11690.187250] Key type .llcrypt registered [11690.234302] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [11690.689596] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11690.709882] alg: No test for adler32 (adler32-zlib) [11691.605426] Lustre: Lustre: Build Version: 2.17.50_30_gc0571ea [11691.733149] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [11693.344180] Key type lgssc registered [11693.896664] Lustre: Echo OBD driver; http://www.lustre.org/ [11698.547521] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11698.574510] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11714.146872] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [11719.264319] LustreError: 391438:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.101@tcp: failed processing log, type 4: rc = -110 [11745.888172] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11750.562813] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [11755.006720] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11759.743679] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11760.117365] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11760.177066] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [11763.609261] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [11766.241095] LustreError: 391464:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11766.262473] LustreError: 391464:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [11769.751318] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11770.081424] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11772.977289] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [11776.414677] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11778.874828] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11785.702351] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11785.703209] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11785.707604] LustreError: Skipped 1 previous similar message [11785.708554] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11785.715061] Lustre: Skipped 1 previous similar message [11785.727776] Lustre: Skipped 1 previous similar message [11789.122140] LustreError: 393712:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [11789.228570] Lustre: server umount lustre-OST0000 complete [11795.944790] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11795.954656] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11795.954982] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11795.957049] Lustre: Skipped 1 previous similar message [11798.443238] LustreError: 393915:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [11798.450181] LustreError: 393915:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [11798.542774] Lustre: server umount lustre-MDT0000 complete [11801.059445] LustreError: 392364:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11801.070366] LustreError: 392364:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11801.289637] LustreError: 391444:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768566045 with bad export cookie 5259521281155877474 [11801.293247] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11801.295303] LustreError: 391444:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [11801.413654] LustreError: 394117:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [11801.418181] LustreError: 394117:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [11801.551443] Lustre: server umount lustre-MDT0001 complete [11808.112132] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [11809.820214] Key type lgssc unregistered [11810.021202] LNet: 394648:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11810.029421] LNetError: 394648:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [11810.053492] LNet: Removed LNI 192.168.203.101@tcp [11810.558440] Key type .llcrypt unregistered [11810.560637] Key type ._llcrypt unregistered [11824.989785] Key type ._llcrypt registered [11824.990775] Key type .llcrypt registered [11825.037497] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [11825.808290] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11825.821823] alg: No test for adler32 (adler32-zlib) [11826.830432] Lustre: Lustre: Build Version: 2.17.50_30_gc0571ea [11826.997584] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [11828.616638] Key type lgssc registered [11829.306845] Lustre: Echo OBD driver; http://www.lustre.org/ [11834.696693] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11834.707671] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11836.024218] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11838.688498] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [11844.985908] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11845.217519] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11848.132792] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [11851.956339] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11855.218710] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11860.292028] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11860.595915] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11861.617288] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [11865.635542] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [11870.256382] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11875.810175] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11875.820919] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11875.836265] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11876.326414] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11876.346815] Lustre: Skipped 1 previous similar message [11876.357258] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11876.367547] Lustre: Skipped 2 previous similar messages [11879.847435] LustreError: 398259:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [11879.949356] Lustre: server umount lustre-MDT0000 complete [11881.446232] LustreError: 396804:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11881.472857] LustreError: 396804:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [11881.689187] LustreError: 398260:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.1@tcp arrived at 1768566125 with bad export cookie 9702102862155708560 [11883.343946] LustreError: 398260:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768566127 with bad export cookie 9702102862155707671 [11883.356936] LustreError: 398260:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [11883.360465] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11886.561982] LustreError: 396804:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11886.563369] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11886.591946] LustreError: 396804:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11886.606706] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11886.609309] Lustre: Skipped 1 previous similar message [11888.613121] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11888.618570] LustreError: 396804:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11888.646320] LustreError: 396804:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11889.463667] LustreError: 398462:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [11889.469501] LustreError: 398462:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [11889.624723] Lustre: server umount lustre-OST0000 complete [11893.731897] LustreError: 396804:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11893.755457] LustreError: 396804:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11895.088442] LustreError: 398815:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [11895.093914] LustreError: 398815:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [11895.310829] Lustre: server umount lustre-MDT0001 complete [11903.276082] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [11905.408788] Key type lgssc unregistered [11905.636827] LNet: 399347:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11905.644860] LNetError: 399347:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [11905.656613] LNet: Removed LNI 192.168.203.101@tcp [11906.277826] Key type .llcrypt unregistered [11906.279684] Key type ._llcrypt unregistered [11924.458245] Key type ._llcrypt registered [11924.460126] Key type .llcrypt registered [11924.533326] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [11925.277138] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11925.290183] alg: No test for adler32 (adler32-zlib) [11926.235184] Lustre: Lustre: Build Version: 2.17.50_30_gc0571ea [11926.397991] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [11928.024727] Key type lgssc registered [11928.893727] Lustre: Echo OBD driver; http://www.lustre.org/ [11934.109197] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11934.131241] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11935.463752] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11937.906903] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [11943.630204] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11943.856347] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11946.448103] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [11950.669698] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11954.024435] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11958.523391] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11958.715987] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11961.778744] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [11962.600964] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [11966.785159] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11971.553840] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [11971.557908] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11971.564786] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11973.091399] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11973.102974] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11974.384691] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.1@tcp (stopping) [11974.392160] Lustre: Skipped 1 previous similar message [11977.095691] LustreError: 402962:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [11977.193594] Lustre: server umount lustre-MDT0001 complete [11978.219386] LustreError: 401882:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11978.241302] LustreError: 401882:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [11982.305161] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11982.312405] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11982.321506] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11983.329778] LustreError: 400723:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11986.745188] LustreError: 403163:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [11986.749796] LustreError: 403163:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [11986.873632] Lustre: server umount lustre-OST0000 complete [11988.451299] LustreError: 400723:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11989.218517] LustreError: 403364:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [11989.221430] LustreError: 403364:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [11989.323598] Lustre: server umount lustre-MDT0000 complete [11997.752958] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [11999.364504] Key type lgssc unregistered [11999.532851] LNet: 404050:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11999.543596] LNetError: 404050:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [11999.567040] LNet: Removed LNI 192.168.203.101@tcp [12000.140757] Key type .llcrypt unregistered [12000.143414] Key type ._llcrypt unregistered [12017.506698] Key type ._llcrypt registered [12017.508737] Key type .llcrypt registered [12017.577230] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_hostid [12028.353681] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [12029.052586] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12029.102369] alg: No test for adler32 (adler32-zlib) [12030.127390] Lustre: Lustre: Build Version: 2.17.50_30_gc0571ea [12030.288504] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [12031.920303] Key type lgssc registered [12032.696106] Lustre: Echo OBD driver; http://www.lustre.org/ [12057.209589] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [12067.435638] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12067.447733] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12068.610068] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12068.643522] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12068.696368] Lustre: lustre-MDT0000: new disk, initializing [12068.759176] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12068.770948] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12071.351832] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [12081.472350] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12081.665074] Lustre: lustre-OST0000: new disk, initializing [12081.668623] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12081.677026] Lustre: Skipped 1 previous similar message [12081.737230] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12083.504991] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [12083.511627] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [12083.534720] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [12086.094578] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [12091.125324] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12102.294694] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12102.366961] Lustre: 407990:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12102.413263] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12102.453131] Lustre: lustre-MDT0001: new disk, initializing [12102.563684] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12102.599446] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [12102.608467] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [12105.460254] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [12111.230851] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [12111.353542] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [12114.077484] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [12114.243151] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [12122.091441] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12122.096165] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12122.109552] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12122.115051] Lustre: Skipped 1 previous similar message [12123.469815] LustreError: 411393:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [12123.634313] Lustre: server umount lustre-MDT0001 complete [12132.321293] LustreError: 408997:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12132.326984] LustreError: 408997:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [12137.448326] LustreError: 407996:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12137.455778] LustreError: 407996:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12140.002129] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12140.007058] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12140.014347] Lustre: Skipped 1 previous similar message [12140.017664] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12140.019761] Lustre: Skipped 1 previous similar message [12142.561531] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12142.561655] LustreError: 408000:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12144.711099] LustreError: 411664:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [12144.715339] LustreError: 411664:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [12144.804446] Lustre: server umount lustre-OST0000 complete [12153.182290] LustreError: 411866:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [12153.188852] LustreError: 411866:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [12153.343658] Lustre: server umount lustre-MDT0000 complete [12161.184934] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [12162.610715] Key type lgssc unregistered [12162.764435] LNet: 412552:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12162.766961] LNetError: 412552:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [12162.775349] LNet: Removed LNI 192.168.203.101@tcp [12163.170788] Key type .llcrypt unregistered [12163.172480] Key type ._llcrypt unregistered [12187.501884] Key type ._llcrypt registered [12187.504218] Key type .llcrypt registered [12187.595289] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [12187.785401] Key type .llcrypt unregistered [12187.788596] Key type ._llcrypt unregistered [12196.848652] Key type ._llcrypt registered [12196.851289] Key type .llcrypt registered [12196.911737] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [12197.353901] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12197.413316] alg: No test for adler32 (adler32-zlib) [12198.344682] Lustre: Lustre: Build Version: 2.17.50_30_gc0571ea [12198.473513] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [12200.128862] Key type lgssc registered [12200.783995] Lustre: Echo OBD driver; http://www.lustre.org/ [12215.023530] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12215.034778] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12216.151585] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12216.169674] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12216.212467] Lustre: lustre-MDT0000: new disk, initializing [12216.252382] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12216.260384] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12217.920541] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [12223.284794] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12223.336858] Lustre: 415862:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12223.364813] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12223.366992] Lustre: Skipped 1 previous similar message [12223.409549] Lustre: lustre-MDT0001: new disk, initializing [12223.461051] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12223.487413] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12223.493805] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12225.270297] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [12229.173269] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12231.186726] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12234.239760] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12234.367205] Lustre: lustre-OST0000: new disk, initializing [12234.370174] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12234.414030] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12236.914957] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [12240.915598] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12241.916668] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12241.922605] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12241.954757] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12247.010475] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12247.011555] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12247.015009] Lustre: Skipped 2 previous similar messages [12247.020973] Lustre: Skipped 2 previous similar messages [12250.757367] LustreError: 418292:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [12250.800158] Lustre: server umount lustre-MDT0000 complete [12252.475524] LustreError: 416638:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768566496 with bad export cookie 16406348884134124773 [12252.479565] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12252.481692] LustreError: 416638:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [12252.558142] LustreError: 418494:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [12252.560558] LustreError: 418494:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [12252.623902] Lustre: server umount lustre-MDT0001 complete [12264.378127] LustreError: 418695:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [12264.386237] LustreError: 418695:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [12264.447522] Lustre: server umount lustre-OST0000 complete [12269.579380] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 07:28:32 (1768566512) [12273.127905] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12273.383342] LustreError: 419512:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12273.393111] LustreError: 419512:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [12273.417920] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12275.411628] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [12278.756859] LustreError: 419512:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12279.887526] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12280.107194] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12282.105475] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [12284.771906] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12286.501290] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12300.774720] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12300.917330] Lustre: Found index 0 for lustre-OST0000, updating log [12300.998333] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12304.793410] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [12308.018947] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12318.446889] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [12320.736940] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12320.740562] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12320.749165] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12321.763711] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12321.766887] Lustre: Skipped 1 previous similar message [12326.832300] LustreError: 421784:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [12326.835318] LustreError: 421784:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [12326.901535] Lustre: server umount lustre-OST0000 complete [12341.728954] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12341.732869] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12341.736586] Lustre: Skipped 1 previous similar message [12341.739024] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12341.740794] Lustre: Skipped 1 previous similar message [12346.308159] LustreError: 422036:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [12346.311489] LustreError: 422036:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [12346.346782] Lustre: server umount lustre-MDT0000 complete [12347.705993] LustreError: 419494:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768566591 with bad export cookie 16406348884134126152 [12347.709492] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12347.714149] LustreError: 419494:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [12347.864494] Lustre: server umount lustre-MDT0001 complete [12351.167766] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 07:29:54 (1768566594) [12359.264429] LNet: 1 local NIs in recovery (showing 1): 192.168.203.101@tcp [12937.838424] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12938.019961] LustreError: 425597:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12938.044950] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12939.573130] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [12942.634119] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12942.758166] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12944.134053] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [12945.912969] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12947.113636] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12960.506788] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12960.613688] Lustre: Found index 0 for lustre-OST0000, updating log [12960.711562] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12962.578315] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [12964.457638] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12965.152768] LustreError: 427902:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [12965.155616] LustreError: 427902:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [12965.190347] Lustre: server umount lustre-OST0000 complete [12968.416776] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12968.419240] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12968.423693] Lustre: Skipped 1 previous similar message [12968.425868] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12968.428418] Lustre: Skipped 2 previous similar messages [12969.441303] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12969.442049] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12970.465771] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12970.468363] Lustre: Skipped 2 previous similar messages [12972.540260] LustreError: 428102:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [12972.542370] LustreError: 428102:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [12972.570114] Lustre: server umount lustre-MDT0000 complete [12973.774973] LustreError: 426312:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768567217 with bad export cookie 16406348884134127223 [12973.777643] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12973.778758] LustreError: 426312:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [12973.894678] Lustre: server umount lustre-MDT0001 complete [12991.484072] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_hostid [12994.663284] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [12999.264169] LNet: 1 local NIs in recovery (showing 1): 192.168.203.101@tcp [13006.008361] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 07:40:49 (1768567249) [13011.475717] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [13012.436826] Key type lgssc unregistered [13022.816951] LNet: 431854:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13022.819369] LNetError: 431854:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [13092.454375] LNet: Removed LNI 192.168.203.101@tcp [13092.762878] Key type .llcrypt unregistered [13092.763987] Key type ._llcrypt unregistered [13099.369766] Key type ._llcrypt registered [13099.371332] Key type .llcrypt registered [13099.409611] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [13099.712384] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13099.743056] alg: No test for adler32 (adler32-zlib) [13100.595894] Lustre: Lustre: Build Version: 2.17.50_30_gc0571ea [13100.675527] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [13102.256152] Key type lgssc registered [13102.596176] Lustre: Echo OBD driver; http://www.lustre.org/ [13104.081566] LNet: Added LNI 192.168.203.101@tcp100 [8/256/0/180] [13104.409177] LNet: Added LNI 192.168.203.101@tcp101 [8/256/0/180] [13104.734244] LNet: Added LNI 192.168.203.101@tcp102 [8/256/0/180] [13105.054091] LNet: Added LNI 192.168.203.101@tcp103 [8/256/0/180] [13105.354981] LNet: Added LNI 192.168.203.101@tcp104 [8/256/0/180] [13105.650832] LNet: Added LNI 192.168.203.101@tcp105 [8/256/0/180] [13105.968100] LNet: Added LNI 192.168.203.101@tcp106 [8/256/0/180] [13106.260384] LNet: Added LNI 192.168.203.101@tcp107 [8/256/0/180] [13106.561982] LNet: Added LNI 192.168.203.101@tcp108 [8/256/0/180] [13106.862025] LNet: Added LNI 192.168.203.101@tcp109 [8/256/0/180] [13107.156300] LNet: Added LNI 192.168.203.101@tcp110 [8/256/0/180] [13107.444774] LNet: Added LNI 192.168.203.101@tcp111 [8/256/0/180] [13107.738952] LNet: Added LNI 192.168.203.101@tcp112 [8/256/0/180] [13108.039858] LNet: Added LNI 192.168.203.101@tcp113 [8/256/0/180] [13108.350317] LNet: Added LNI 192.168.203.101@tcp114 [8/256/0/180] [13108.637136] LNet: Added LNI 192.168.203.101@tcp115 [8/256/0/180] [13108.929857] LNet: Added LNI 192.168.203.101@tcp116 [8/256/0/180] [13109.217368] LNet: Added LNI 192.168.203.101@tcp117 [8/256/0/180] [13109.518451] LNet: Added LNI 192.168.203.101@tcp118 [8/256/0/180] [13109.836705] LNet: Added LNI 192.168.203.101@tcp119 [8/256/0/180] [13110.144078] LNet: Added LNI 192.168.203.101@tcp120 [8/256/0/180] [13110.463552] LNet: Added LNI 192.168.203.101@tcp121 [8/256/0/180] [13110.764400] LNet: Added LNI 192.168.203.101@tcp122 [8/256/0/180] [13111.066075] LNet: Added LNI 192.168.203.101@tcp123 [8/256/0/180] [13111.369776] LNet: Added LNI 192.168.203.101@tcp124 [8/256/0/180] [13111.652881] LNet: Added LNI 192.168.203.101@tcp125 [8/256/0/180] [13111.948271] LNet: Added LNI 192.168.203.101@tcp126 [8/256/0/180] [13112.239313] LNet: Added LNI 192.168.203.101@tcp127 [8/256/0/180] [13112.550870] LNet: Added LNI 192.168.203.101@tcp128 [8/256/0/180] [13112.842301] LNet: Added LNI 192.168.203.101@tcp129 [8/256/0/180] [13113.129494] LNet: Added LNI 192.168.203.101@tcp130 [8/256/0/180] [13113.415157] LNet: Added LNI 192.168.203.101@tcp131 [8/256/0/180] [13113.714373] LNet: Added LNI 192.168.203.101@tcp132 [8/256/0/180] [13114.006993] LNet: Added LNI 192.168.203.101@tcp133 [8/256/0/180] [13114.294332] LNet: Added LNI 192.168.203.101@tcp134 [8/256/0/180] [13114.587494] LNet: Added LNI 192.168.203.101@tcp135 [8/256/0/180] [13116.701898] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13116.707196] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13117.753750] Lustre: 434839:0:(lustre_peer.c:105:class_add_uuid()) 192.168.203.101@tcp: can't add NID 192.168.203.101@tcp130: rc = -75 [13117.822860] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13117.834075] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13117.864210] Lustre: lustre-MDT0000: new disk, initializing [13117.892281] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13117.898087] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13118.822069] LNetError: 434861:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.101 [13118.825673] LNetError: Protocol error connecting to 192.168.203.101@tcp112 at host 192.168.203.101:988: is it running a compatible version of Lustre? [13119.093174] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [13122.996585] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13123.044818] Lustre: 434857:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13123.068437] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13123.070060] Lustre: Skipped 1 previous similar message [13123.103304] Lustre: lustre-MDT0001: new disk, initializing [13123.114755] Lustre: 435641:0:(lustre_peer.c:105:class_add_uuid()) 192.168.203.101@tcp: can't add NID 192.168.203.101@tcp130: rc = -75 [13123.117747] Lustre: 435641:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [13123.154117] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13123.166684] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13123.169767] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13124.403527] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [13125.874907] Lustre: 435989:0:(lustre_peer.c:105:class_add_uuid()) 192.168.203.101@tcp: can't add NID 192.168.203.101@tcp130: rc = -75 [13125.877968] Lustre: 435989:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [13127.441478] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13128.585476] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13130.721518] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13130.821198] Lustre: lustre-OST0000: new disk, initializing [13130.822875] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13130.855958] Lustre: 436610:0:(lustre_peer.c:105:class_add_uuid()) 192.168.203.101@tcp: can't add NID 192.168.203.101@tcp130: rc = -75 [13130.858187] Lustre: 436610:0:(lustre_peer.c:105:class_add_uuid()) Skipped 5 previous similar messages [13130.876155] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13132.710195] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [13134.715662] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13134.718540] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13134.760316] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13135.676632] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13139.937464] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13139.939073] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13139.940392] LustreError: Skipped 1 previous similar message [13139.944561] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13139.946824] Lustre: Skipped 1 previous similar message [13143.916881] LustreError: 437442:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [13143.950689] Lustre: server umount lustre-OST0000 complete [13148.640837] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13148.643257] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13148.647078] Lustre: Skipped 1 previous similar message [13148.648736] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13150.177125] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13150.177588] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13152.929144] LNetError: 432303:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.101 [13152.932729] LNetError: Protocol error connecting to 192.168.203.101@tcp112 at host 192.168.203.101:988: is it running a compatible version of Lustre? [13155.297235] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13155.300347] Lustre: Skipped 2 previous similar messages [13159.392124] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13159.424872] LustreError: 437644:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [13159.426855] LustreError: 437644:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [13159.453038] Lustre: server umount lustre-MDT0000 complete [13160.646037] LustreError: 434848:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768567404 with bad export cookie 9341468426677763543 [13160.649063] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13160.651100] LustreError: 434848:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [13160.709825] LustreError: 437846:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [13160.712202] LustreError: 437846:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [13160.764842] Lustre: server umount lustre-MDT0001 complete [13164.099170] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [13164.194383] LNetError: 432304:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.101 [13164.199183] LNetError: Protocol error connecting to 192.168.203.101@tcp112 at host 192.168.203.101:988: is it running a compatible version of Lustre? [13165.098563] Key type lgssc unregistered [13165.257409] LNet: 438385:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13165.260564] LNetError: 438385:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [13165.262639] LNet: Removed LNI 192.168.203.101@tcp [13166.304135] LNet: Removed LNI 192.168.203.101@tcp100 [13166.305895] LNetError: 438385:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [13166.308632] LNetError: 438385:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [13166.312994] LNet: Removed LNI 192.168.203.101@tcp101 [13167.328134] LNet: Removed LNI 192.168.203.101@tcp102 [13167.329821] LNetError: 438385:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [13167.332338] LNetError: 438385:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [13167.335029] LNet: Removed LNI 192.168.203.101@tcp103 [13168.352113] LNet: Removed LNI 192.168.203.101@tcp104 [13169.376125] LNet: Removed LNI 192.168.203.101@tcp105 [13169.377503] LNetError: 438385:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [13169.379412] LNetError: 438385:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [13169.381266] LNet: Removed LNI 192.168.203.101@tcp106 [13170.400115] LNet: Removed LNI 192.168.203.101@tcp107 [13171.424157] LNet: Removed LNI 192.168.203.101@tcp108 [13172.448111] LNet: Removed LNI 192.168.203.101@tcp109 [13173.472112] LNet: Removed LNI 192.168.203.101@tcp110 [13173.474098] LNetError: 438385:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [13173.476440] LNetError: 438385:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [13173.478674] LNet: Removed LNI 192.168.203.101@tcp111 [13174.496114] LNet: Removed LNI 192.168.203.101@tcp112 [13175.520084] LNet: Removed LNI 192.168.203.101@tcp113 [13176.544125] LNet: Removed LNI 192.168.203.101@tcp114 [13177.568130] LNet: Removed LNI 192.168.203.101@tcp115 [13178.592120] LNet: Removed LNI 192.168.203.101@tcp116 [13179.616121] LNet: Removed LNI 192.168.203.101@tcp117 [13179.617253] LNet: Removed LNI 192.168.203.101@tcp118 [13179.618330] LNet: Removed LNI 192.168.203.101@tcp119 [13179.619372] LNet: Removed LNI 192.168.203.101@tcp120 [13179.620313] LNet: Removed LNI 192.168.203.101@tcp121 [13179.621297] LNet: Removed LNI 192.168.203.101@tcp122 [13179.622435] LNet: Removed LNI 192.168.203.101@tcp123 [13179.623635] LNet: Removed LNI 192.168.203.101@tcp124 [13179.624673] LNet: Removed LNI 192.168.203.101@tcp125 [13179.625715] LNet: Removed LNI 192.168.203.101@tcp126 [13179.626726] LNet: Removed LNI 192.168.203.101@tcp127 [13179.627712] LNet: Removed LNI 192.168.203.101@tcp128 [13179.628735] LNet: Removed LNI 192.168.203.101@tcp129 [13179.629782] LNet: Removed LNI 192.168.203.101@tcp130 [13179.630830] LNet: Removed LNI 192.168.203.101@tcp131 [13179.631850] LNet: Removed LNI 192.168.203.101@tcp132 [13179.632919] LNet: Removed LNI 192.168.203.101@tcp133 [13179.633960] LNet: Removed LNI 192.168.203.101@tcp134 [13179.641424] LNet: Removed LNI 192.168.203.101@tcp135 [13179.916813] Key type .llcrypt unregistered [13179.917770] Key type ._llcrypt unregistered [13187.431269] Key type ._llcrypt registered [13187.432228] Key type .llcrypt registered [13187.466665] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_hostid [13193.082233] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [13193.440477] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13193.464856] alg: No test for adler32 (adler32-zlib) [13194.302597] Lustre: Lustre: Build Version: 2.17.50_30_gc0571ea [13194.379554] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [13195.952220] Key type lgssc registered [13196.340117] Lustre: Echo OBD driver; http://www.lustre.org/ [13207.100512] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 07:44:10 (1768567450) [13212.957097] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [13213.975126] Key type lgssc unregistered [13214.109590] LNet: 442401:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13214.111877] LNetError: 442401:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [13214.120247] LNet: Removed LNI 192.168.203.101@tcp [13214.411305] Key type .llcrypt unregistered [13214.413046] Key type ._llcrypt unregistered [13220.785531] Key type ._llcrypt registered [13220.786956] Key type .llcrypt registered [13220.827297] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [13221.257667] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13221.314964] alg: No test for adler32 (adler32-zlib) [13222.172313] Lustre: Lustre: Build Version: 2.17.50_30_gc0571ea [13222.257997] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [13223.848150] Key type lgssc registered [13224.248051] Lustre: Echo OBD driver; http://www.lustre.org/ [13227.186243] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13227.190655] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13228.280351] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13228.291280] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13228.327120] Lustre: lustre-MDT0000: new disk, initializing [13228.350049] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13228.356362] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13229.604626] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [13233.595365] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13233.633573] Lustre: 443572:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13233.655090] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13233.657359] Lustre: Skipped 1 previous similar message [13233.690534] Lustre: lustre-MDT0001: new disk, initializing [13233.720566] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13233.733842] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13233.736933] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13234.945268] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [13237.989933] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13239.168935] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13241.402648] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13241.508183] Lustre: lustre-OST0000: new disk, initializing [13241.510058] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13241.536535] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13243.260820] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [13244.918331] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13244.920878] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13244.951255] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13246.120832] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13247.111802] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing lnet_if_list [13248.592716] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [13248.604067] LNet: Added LNI 192.168.203.101@tcp10 [8/256/0/180] [13250.174377] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [13250.184599] LNet: Added LNI 192.168.203.101@tcp11 [8/256/0/180] [13251.302155] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [13251.311202] LNet: Added LNI 192.168.203.101@tcp12 [8/256/0/180] [13252.504799] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [13252.514330] LNet: Added LNI 192.168.203.101@tcp13 [8/256/0/180] [13253.730555] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [13253.740214] LNet: Added LNI 192.168.203.101@tcp14 [8/256/0/180] [13255.046241] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [13255.056903] LNet: Added LNI 192.168.203.101@tcp15 [8/256/0/180] [13256.393817] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [13256.405409] LNet: Added LNI 192.168.203.101@tcp16 [8/256/0/180] [13257.754473] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [13257.764224] LNet: Added LNI 192.168.203.101@tcp17 [8/256/0/180] [13259.151685] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [13259.161801] LNet: Added LNI 192.168.203.101@tcp18 [8/256/0/180] [13260.625673] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [13260.639799] LNet: Added LNI 192.168.203.101@tcp19 [8/256/0/180] [13262.115486] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [13262.128539] LNet: Added LNI 192.168.203.101@tcp20 [8/256/0/180] [13263.564521] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [13263.575489] LNet: Added LNI 192.168.203.101@tcp21 [8/256/0/180] [13265.108683] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [13265.120519] LNet: Added LNI 192.168.203.101@tcp22 [8/256/0/180] [13266.616253] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [13266.627768] LNet: Added LNI 192.168.203.101@tcp23 [8/256/0/180] [13268.173444] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [13268.185977] LNet: Added LNI 192.168.203.101@tcp24 [8/256/0/180] [13269.718795] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [13269.730186] LNet: Added LNI 192.168.203.101@tcp25 [8/256/0/180] [13271.326345] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [13271.337355] LNet: Added LNI 192.168.203.101@tcp26 [8/256/0/180] [13273.037769] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [13273.049374] LNet: Added LNI 192.168.203.101@tcp27 [8/256/0/180] [13273.055173] LNetError: 442834:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.101 [13273.058588] LNetError: Protocol error connecting to 192.168.203.101@tcp26 at host 192.168.203.101:988: is it running a compatible version of Lustre? [13274.779217] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [13274.790219] LNet: Added LNI 192.168.203.101@tcp28 [8/256/0/180] [13274.793906] LNetError: 442831:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.101 [13274.798685] LNetError: Protocol error connecting to 192.168.203.101@tcp26 at host 192.168.203.101:988: is it running a compatible version of Lustre? [13276.584367] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [13276.593041] LNet: Added LNI 192.168.203.101@tcp29 [8/256/0/180] [13276.596725] LNetError: 442832:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.101 [13276.601648] LNetError: Protocol error connecting to 192.168.203.101@tcp28 at host 192.168.203.101:988: is it running a compatible version of Lustre? [13277.536191] LNet: 2 local NIs in recovery (showing 2): 192.168.203.101@tcp28, 192.168.203.101@tcp29 [13278.322454] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [13278.331736] LNet: Added LNI 192.168.203.101@tcp30 [8/256/0/180] [13279.585021] LNetError: 442832:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.101 [13279.587305] LNetError: 442832:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 3 previous similar messages [13279.589945] LNetError: Protocol error connecting to 192.168.203.101@tcp28 at host 192.168.203.101:988: is it running a compatible version of Lustre? [13279.594333] LNetError: Skipped 3 previous similar messages [13280.154552] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [13280.165291] LNet: Added LNI 192.168.203.101@tcp31 [8/256/0/180] [13282.014310] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [13282.027521] LNet: Added LNI 192.168.203.101@tcp32 [8/256/0/180] [13283.682191] LNetError: 442834:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.101 [13283.685624] LNetError: 442834:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 3 previous similar messages [13283.687779] LNetError: Protocol error connecting to 192.168.203.101@tcp28 at host 192.168.203.101:988: is it running a compatible version of Lustre? [13283.691267] LNetError: Skipped 3 previous similar messages [13283.988916] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [13284.002593] LNet: Added LNI 192.168.203.101@tcp33 [8/256/0/180] [13285.964533] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [13285.976183] LNet: Added LNI 192.168.203.101@tcp34 [8/256/0/180] [13287.936900] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [13287.947051] LNet: Added LNI 192.168.203.101@tcp35 [8/256/0/180] [13289.844588] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [13289.854729] LNet: Added LNI 192.168.203.101@tcp36 [8/256/0/180] [13291.683495] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [13291.692424] LNet: Added LNI 192.168.203.101@tcp37 [8/256/0/180] [13291.875361] LNetError: 442834:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.101 [13291.878922] LNetError: 442834:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 15 previous similar messages [13291.881271] LNetError: Protocol error connecting to 192.168.203.101@tcp28 at host 192.168.203.101:988: is it running a compatible version of Lustre? [13291.883977] LNetError: Skipped 15 previous similar messages [13293.617643] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [13293.627679] LNet: Added LNI 192.168.203.101@tcp38 [8/256/0/180] [13295.751896] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [13295.762209] LNet: Added LNI 192.168.203.101@tcp39 [8/256/0/180] [13297.921901] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [13297.934825] LNet: Added LNI 192.168.203.101@tcp40 [8/256/0/180] [13300.289862] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [13300.301036] LNet: Added LNI 192.168.203.101@tcp41 [8/256/0/180] [13302.511398] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [13302.521201] LNet: Added LNI 192.168.203.101@tcp42 [8/256/0/180] [13304.849668] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [13304.859086] LNet: Added LNI 192.168.203.101@tcp43 [8/256/0/180] [13307.123310] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [13307.132898] LNet: Added LNI 192.168.203.101@tcp44 [8/256/0/180] [13308.258184] LNetError: 442834:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.101 [13308.260985] LNetError: 442834:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 30 previous similar messages [13308.263365] LNetError: Protocol error connecting to 192.168.203.101@tcp36 at host 192.168.203.101:988: is it running a compatible version of Lustre? [13308.266155] LNetError: Skipped 30 previous similar messages [13309.361471] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [13309.372749] LNet: Added LNI 192.168.203.101@tcp45 [8/256/0/180] [13311.653137] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [13311.663577] LNet: Added LNI 192.168.203.101@tcp46 [8/256/0/180] [13313.988686] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [13313.999651] LNet: Added LNI 192.168.203.101@tcp47 [8/256/0/180] [13316.499595] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [13316.511436] LNet: Added LNI 192.168.203.101@tcp48 [8/256/0/180] [13318.953988] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [13318.962739] LNet: Added LNI 192.168.203.101@tcp49 [8/256/0/180] [13321.426259] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [13321.439527] LNet: Added LNI 192.168.203.101@tcp50 [8/256/0/180] [13331.939333] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13331.940389] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13331.941961] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13331.944479] LustreError: Skipped 1 previous similar message [13331.949659] Lustre: Skipped 1 previous similar message [13334.444123] LustreError: 467740:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [13334.484481] Lustre: server umount lustre-OST0000 complete [13336.032694] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13336.034806] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13336.038500] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13336.040797] Lustre: Skipped 1 previous similar message [13336.928144] LNet: 7 local NIs in recovery (showing 7): 192.168.203.101@tcp41, 192.168.203.101@tcp47, 192.168.203.101@tcp46, 192.168.203.101@tcp45, 192.168.203.101@tcp36, 192.168.203.101@tcp33, 192.168.203.101@tcp28 [13337.057122] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13337.057810] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13341.024852] LNetError: 442832:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.101 [13341.027895] LNetError: 442832:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 34 previous similar messages [13341.030133] LNetError: Protocol error connecting to 192.168.203.101@tcp37 at host 192.168.203.101:988: is it running a compatible version of Lustre? [13341.033008] LNetError: Skipped 34 previous similar messages [13341.760776] LustreError: 467941:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [13341.762898] LustreError: 467941:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [13341.785612] Lustre: server umount lustre-MDT0000 complete [13342.956074] LustreError: 444352:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768567587 with bad export cookie 1341658656204749484 [13342.959319] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13342.961365] LustreError: 444352:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [13343.022827] LustreError: 468142:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [13343.024940] LustreError: 468142:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [13343.071762] Lustre: server umount lustre-MDT0001 complete [13346.519759] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [13347.562191] Key type lgssc unregistered [13347.725701] LNet: 468682:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13347.729530] LNetError: 468682:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [13347.732097] LNet: Removed LNI 192.168.203.101@tcp [13348.768145] LNet: Removed LNI 192.168.203.101@tcp10 [13348.770142] LNetError: 468682:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [13348.773210] LNetError: 468682:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [13348.776276] LNet: Removed LNI 192.168.203.101@tcp11 [13349.792197] LNet: Removed LNI 192.168.203.101@tcp12 [13349.793960] LNetError: 468682:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [13349.797509] LNetError: 468682:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [13349.800294] LNet: Removed LNI 192.168.203.101@tcp13 [13350.816086] LNet: Removed LNI 192.168.203.101@tcp14 [13351.840147] LNet: Removed LNI 192.168.203.101@tcp15 [13351.842109] LNetError: 468682:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [13351.844571] LNetError: 468682:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [13351.847493] LNet: Removed LNI 192.168.203.101@tcp16 [13352.864116] LNet: Removed LNI 192.168.203.101@tcp17 [13353.888322] LNet: Removed LNI 192.168.203.101@tcp18 [13354.912166] LNet: Removed LNI 192.168.203.101@tcp19 [13355.936106] LNet: Removed LNI 192.168.203.101@tcp20 [13355.937744] LNetError: 468682:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [13355.939635] LNetError: 468682:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [13355.943257] LNet: Removed LNI 192.168.203.101@tcp21 [13356.960106] LNet: Removed LNI 192.168.203.101@tcp22 [13357.984118] LNet: Removed LNI 192.168.203.101@tcp23 [13359.008174] LNet: Removed LNI 192.168.203.101@tcp24 [13360.032107] LNet: Removed LNI 192.168.203.101@tcp25 [13361.056117] LNet: Removed LNI 192.168.203.101@tcp26 [13362.080149] LNet: Removed LNI 192.168.203.101@tcp27 [13362.081510] LNet: Removed LNI 192.168.203.101@tcp28 [13363.104184] LNet: Removed LNI 192.168.203.101@tcp29 [13364.128149] LNet: Removed LNI 192.168.203.101@tcp30 [13364.129566] LNetError: 468682:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [13364.131827] LNetError: 468682:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 9 previous similar messages [13364.134182] LNet: Removed LNI 192.168.203.101@tcp31 [13365.152099] LNet: Removed LNI 192.168.203.101@tcp32 [13365.153494] LNet: Removed LNI 192.168.203.101@tcp33 [13366.176147] LNet: Removed LNI 192.168.203.101@tcp34 [13367.200171] LNet: Removed LNI 192.168.203.101@tcp35 [13367.201546] LNet: Removed LNI 192.168.203.101@tcp36 [13368.224128] LNet: Removed LNI 192.168.203.101@tcp37 [13369.248156] LNet: Removed LNI 192.168.203.101@tcp38 [13370.272125] LNet: Removed LNI 192.168.203.101@tcp39 [13371.296137] LNet: Removed LNI 192.168.203.101@tcp40 [13371.297440] LNet: Removed LNI 192.168.203.101@tcp41 [13372.320097] LNet: Removed LNI 192.168.203.101@tcp42 [13373.344120] LNet: Removed LNI 192.168.203.101@tcp43 [13374.368144] LNet: Removed LNI 192.168.203.101@tcp44 [13375.392194] LNet: Removed LNI 192.168.203.101@tcp45 [13375.394332] LNet: Removed LNI 192.168.203.101@tcp46 [13375.396145] LNet: Removed LNI 192.168.203.101@tcp47 [13375.397781] LNet: Removed LNI 192.168.203.101@tcp48 [13375.399715] LNet: Removed LNI 192.168.203.101@tcp49 [13375.407392] LNet: Removed LNI 192.168.203.101@tcp50 [13375.697054] Key type .llcrypt unregistered [13375.698345] Key type ._llcrypt unregistered [13382.979792] Key type ._llcrypt registered [13382.980624] Key type .llcrypt registered [13383.013921] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [13383.394423] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13383.403025] alg: No test for adler32 (adler32-zlib) [13384.254409] Lustre: Lustre: Build Version: 2.17.50_30_gc0571ea [13384.325698] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [13385.896162] Key type lgssc registered [13386.191919] Lustre: Echo OBD driver; http://www.lustre.org/ [13388.636362] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13388.643773] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13389.820627] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13391.099597] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [13393.870918] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13393.979747] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13395.213397] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [13396.959369] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13398.164214] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13400.223805] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13400.326035] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13402.008554] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [13403.650850] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13422.879129] loop0: detected capacity change from 0 to 204800000 [13422.933065] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13422.936249] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13422.940897] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13424.808486] loop0: detected capacity change from 0 to 204800000 [13424.865074] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13424.869419] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13424.873113] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13426.478520] loop0: detected capacity change from 0 to 204800000 [13428.244774] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13428.287077] Lustre: fs15246-OST0000: new disk, initializing [13428.288646] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [13428.304525] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13430.002178] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [13433.660712] loop1: detected capacity change from 0 to 204800000 [13435.425128] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13435.457133] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [13435.476017] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [13435.507725] Lustre: fs15246-MDT0000: new disk, initializing [13435.551739] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13435.560350] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13435.593302] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [13436.641745] Lustre: 474285:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@000000004cf2a3e3 x1854477569129728/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1768567696 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [13436.710411] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [13445.728199] LustreError: 474285:0:(fid_request.c:213:seq_client_alloc_seq()) cli-cli-fs15246-OST0000-osc-MDT0000: Cannot allocate new meta-sequence: rc = -5 [13445.730785] LustreError: 474285:0:(fid_request.c:255:seq_client_get_seq()) cli-cli-fs15246-OST0000-osc-MDT0000: Can't allocate new sequence: rc = -5 [13445.733314] LustreError: 474285:0:(osp_precreate.c:459:osp_precreate_rollover_new_seq()) fs15246-OST0000-osc-MDT0000: alloc fid error: rc = -5 [13448.164302] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13448.168803] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [13449.725571] LustreError: 475410:0:(obd_class.h:479:obd_check_dev()) Device 30 not setup [13449.751586] Lustre: server umount fs15246-MDT0000 complete [13452.972327] LustreError: 475912:0:(obd_class.h:479:obd_check_dev()) Device 32 not setup [13452.974521] LustreError: 475912:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [13452.992612] Lustre: server umount fs15246-OST0000 complete [13456.865087] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13456.867801] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13456.872839] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13458.401994] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13458.406296] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13462.511119] LustreError: 476413:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [13462.514658] LustreError: 476413:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [13462.557923] Lustre: server umount lustre-OST0000 complete [13465.568942] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13465.572799] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13465.579530] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13465.582251] Lustre: Skipped 1 previous similar message [13473.761090] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13473.763537] Lustre: Skipped 3 previous similar messages [13478.368121] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13478.404442] LustreError: 476615:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [13478.406321] LustreError: 476615:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [13478.437818] Lustre: server umount lustre-MDT0000 complete [13478.881176] LustreError: 472066:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13478.887948] LustreError: 472066:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [13479.699236] LustreError: 469885:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768567723 with bad export cookie 10098992640446701613 [13479.701644] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13479.704591] LustreError: 469885:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [13479.818478] Lustre: server umount lustre-MDT0001 complete [13483.630993] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [13484.702944] Key type lgssc unregistered [13484.848329] LNet: 477348:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13484.850754] LNetError: 477348:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [13484.857321] LNet: Removed LNI 192.168.203.101@tcp [13485.185737] Key type .llcrypt unregistered [13485.187733] Key type ._llcrypt unregistered [13492.536293] Key type ._llcrypt registered [13492.538134] Key type .llcrypt registered [13492.578465] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [13492.996201] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13493.018957] alg: No test for adler32 (adler32-zlib) [13493.871630] Lustre: Lustre: Build Version: 2.17.50_30_gc0571ea [13493.954475] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [13495.544176] Key type lgssc registered [13495.919981] Lustre: Echo OBD driver; http://www.lustre.org/ [13508.116483] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_hostid [13511.191310] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [13522.142419] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13522.147285] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13523.222593] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13523.231489] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13523.262222] Lustre: lustre-MDT0000: new disk, initializing [13523.282085] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13523.288048] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13524.470709] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [13528.337534] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13528.372619] Lustre: 481728:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13528.391825] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13528.393290] Lustre: Skipped 1 previous similar message [13528.428101] Lustre: lustre-MDT0001: new disk, initializing [13528.457286] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13528.467754] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13528.471145] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13529.620900] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [13532.624967] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13533.838386] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13536.086811] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13536.191962] Lustre: lustre-OST0000: new disk, initializing [13536.193779] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13536.221692] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13538.098714] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [13538.746875] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13538.751190] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13538.779959] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13541.221861] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13542.611488] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [13542.684338] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [13544.023662] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [13544.097632] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [13549.027803] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13549.028648] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13549.034697] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13549.037774] Lustre: Skipped 1 previous similar message [13554.147387] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13554.149601] Lustre: Skipped 1 previous similar message [13559.264154] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13559.266672] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13559.269754] Lustre: Skipped 1 previous similar message [13559.288629] LustreError: 484660:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [13559.328994] Lustre: server umount lustre-OST0000 complete [13564.385947] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13564.385959] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13564.389738] Lustre: Skipped 1 previous similar message [13564.390270] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13564.391657] LustreError: Skipped 1 previous similar message [13566.655792] LustreError: 484865:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [13566.657743] LustreError: 484865:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [13566.686824] Lustre: server umount lustre-MDT0000 complete [13567.949410] LustreError: 481719:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768567812 with bad export cookie 3402685495512795191 [13567.952716] LustreError: 481719:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [13567.953473] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13568.023646] LustreError: 485067:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [13568.027161] LustreError: 485067:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [13568.086490] Lustre: server umount lustre-MDT0001 complete [13575.927323] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_hostid [13579.058812] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [13589.159596] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 13251 sec ========= 07:50:32 (1768567832) [13589.696219] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 07:50:32 (1768567832) === [13590.801598] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 07:50:34 (1768567834) === [13603.943080] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [13605.025477] Key type lgssc unregistered [13605.147379] LNet: 489904:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13605.149718] LNetError: 489904:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [13605.157371] LNet: Removed LNI 192.168.203.101@tcp [13605.441191] Key type .llcrypt unregistered [13605.442965] Key type ._llcrypt unregistered