[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 2798903870 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2439 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22D5 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 002295 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2349 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23D9 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2411 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22d5-0xbffe2348] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22d4] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2349-0xbffe23d8] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23d9-0xbffe2410] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2411-0xbffe2438] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.002000] x2apic enabled [ 0.002000] Switched APIC routing to physical x2apic. [ 0.002000] kvm-guest: setup PV IPIs [ 0.002000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.002000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.002000] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.002000] pid_max: default: 32768 minimum: 301 [ 0.002000] LSM: Security Framework initializing [ 0.002000] Yama: becoming mindful. [ 0.002000] SELinux: Initializing. [ 0.002000] *** VALIDATE selinux *** [ 0.018238] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.031735] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.033759] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.034120] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.035132] *** VALIDATE tmpfs *** [ 0.039216] *** VALIDATE proc *** [ 0.043049] *** VALIDATE cgroup *** [ 0.044015] *** VALIDATE cgroup2 *** [ 0.047055] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.049094] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.050013] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.051031] Spectre V2 : User space: Vulnerable [ 0.052009] Speculative Store Bypass: Vulnerable [ 0.056294] debug: unmapping init [mem 0xffffffffb3259000-0xffffffffb3260fff] [ 0.058422] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.059941] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.060021] ... version: 2 [ 0.061021] ... bit width: 48 [ 0.062017] ... generic registers: 4 [ 0.063010] ... value mask: 0000ffffffffffff [ 0.064012] ... max period: 00007fffffffffff [ 0.065015] ... fixed-purpose events: 3 [ 0.066217] ... event mask: 000000070000000f [ 0.067577] rcu: Hierarchical SRCU implementation. [ 0.070298] smp: Bringing up secondary CPUs ... [ 0.071890] x86: Booting SMP configuration: [ 0.072030] .... node #0, CPUs: #1 #2 #3 [ 0.083293] smp: Brought up 1 node, 4 CPUs [ 0.085009] smpboot: Max logical packages: 1 [ 0.086012] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.112386] node 0 deferred pages initialised in 24ms [ 0.117009] devtmpfs: initialized [ 0.121082] x86/mm: Memory block size: 128MB [ 0.128490] gcov: version magic: 0x41383552 [ 0.130541] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.131076] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.134329] pinctrl core: initialized pinctrl subsystem [ 0.136293] [ 0.137006] ************************************************************* [ 0.140010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.143023] ** ** [ 0.149013] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.154015] ** ** [ 0.157013] ** This means that this kernel is built to expose internal ** [ 0.161017] ** IOMMU data structures, which may compromise security on ** [ 0.164010] ** your system. ** [ 0.166145] ** ** [ 0.169013] ** If you see this message and you are not debugging the ** [ 0.173014] ** kernel, report this immediately to your vendor! ** [ 0.176012] ** ** [ 0.179010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.181013] ************************************************************* [ 0.184817] NET: Registered protocol family 16 [ 0.186648] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.189052] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.192197] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.196011] cpuidle: using governor menu [ 0.198000] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.201080] PCI: Using configuration type 1 for base access [ 0.203127] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.212166] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.214024] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.219055] cryptd: max_cpu_qlen set to 1000 [ 0.221242] ACPI: Added _OSI(Module Device) [ 0.223031] ACPI: Added _OSI(Processor Device) [ 0.224014] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.226091] ACPI: Added _OSI(Processor Aggregator Device) [ 0.232183] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.240581] ACPI: Interpreter enabled [ 0.242081] ACPI: PM: (supports S0 S3 S4 S5) [ 0.244051] ACPI: Using IOAPIC for interrupt routing [ 0.246127] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.253580] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.274862] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.277042] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.280019] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.284081] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.293658] acpiphp: Slot [2] registered [ 0.298172] acpiphp: Slot [5] registered [ 0.302145] acpiphp: Slot [6] registered [ 0.304150] acpiphp: Slot [7] registered [ 0.346726] acpiphp: Slot [8] registered [ 0.358186] acpiphp: Slot [9] registered [ 0.365220] acpiphp: Slot [10] registered [ 0.375102] acpiphp: Slot [3] registered [ 0.381539] acpiphp: Slot [4] registered [ 0.388289] acpiphp: Slot [11] registered [ 0.396988] acpiphp: Slot [12] registered [ 0.407126] acpiphp: Slot [13] registered [ 0.414339] acpiphp: Slot [14] registered [ 0.423242] acpiphp: Slot [15] registered [ 0.429967] acpiphp: Slot [16] registered [ 0.436435] acpiphp: Slot [17] registered [ 0.443205] acpiphp: Slot [18] registered [ 0.448462] acpiphp: Slot [19] registered [ 0.454121] acpiphp: Slot [20] registered [ 0.460120] acpiphp: Slot [21] registered [ 0.466744] acpiphp: Slot [22] registered [ 0.471118] acpiphp: Slot [23] registered [ 0.476166] acpiphp: Slot [24] registered [ 0.482780] acpiphp: Slot [25] registered [ 0.485112] acpiphp: Slot [26] registered [ 0.487093] acpiphp: Slot [27] registered [ 0.489146] acpiphp: Slot [28] registered [ 0.491208] acpiphp: Slot [29] registered [ 0.493106] acpiphp: Slot [30] registered [ 0.495177] acpiphp: Slot [31] registered [ 0.497245] PCI host bridge to bus 0000:00 [ 0.500023] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.502023] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.507033] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.512022] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.516028] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.520023] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.522260] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.527754] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.532758] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.549013] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.555052] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.559025] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.562099] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.565018] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.567657] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.571031] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.574045] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.579949] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.584826] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.599015] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.604928] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.610396] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.709017] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.719017] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.788028] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.802010] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.811015] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.822016] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.868015] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.878000] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.878000] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.878000] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.886015] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.898000] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.898000] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.898000] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.898000] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.909679] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.915012] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.920014] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.931018] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.939038] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.944012] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.950012] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.963018] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.999270] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 1.004280] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 1.009381] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 1.013365] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 1.016182] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 1.028072] iommu: Default domain type: Passthrough [ 1.029470] SCSI subsystem initialized [ 1.030152] ACPI: bus type USB registered [ 1.031838] usbcore: registered new interface driver usbfs [ 1.032102] usbcore: registered new interface driver hub [ 1.033104] usbcore: registered new device driver usb [ 1.034204] pps_core: LinuxPPS API ver. 1 registered [ 1.035011] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 1.036056] PTP clock support registered [ 1.041071] EDAC MC: Ver: 3.0.0 [ 1.042000] PCI: Using ACPI for IRQ routing [ 1.044718] NetLabel: Initializing [ 1.047012] NetLabel: domain hash size = 128 [ 1.050011] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 1.054220] NetLabel: unlabeled traffic allowed by default [ 1.060045] vgaarb: loaded [ 1.062735] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 1.068012] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 1.076985] clocksource: Switched to clocksource kvm-clock [ 1.376930] VFS: Disk quotas dquot_6.6.0 [ 1.384687] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.389826] *** VALIDATE ramfs *** [ 1.394690] *** VALIDATE hugetlbfs *** [ 1.398699] pnp: PnP ACPI init [ 1.403915] pnp: PnP ACPI: found 6 devices [ 1.422986] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.425815] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.427619] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.429318] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.431198] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.433024] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.435248] NET: Registered protocol family 2 [ 1.437390] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.441759] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.445594] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.450696] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.454216] TCP: Hash tables configured (established 65536 bind 65536) [ 1.457116] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.460751] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.463872] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.468109] NET: Registered protocol family 1 [ 1.471561] RPC: Registered named UNIX socket transport module. [ 1.474286] RPC: Registered udp transport module. [ 1.492104] RPC: Registered tcp transport module. [ 1.500492] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.504638] NET: Registered protocol family 44 [ 1.506621] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.509967] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.513160] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.518250] PCI: CLS 0 bytes, default 64 [ 1.524203] Unpacking initramfs... [ 6.526214] debug: unmapping init [mem 0xffff8c117cc54000-0xffff8c117ffbffff] [ 6.539503] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 6.546199] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 6.558912] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 8.796616] Initialise system trusted keyrings [ 8.798828] Key type blacklist registered [ 8.801424] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 8.812595] zbud: loaded [ 8.818409] *** VALIDATE nfs *** [ 8.820040] *** VALIDATE nfs4 *** [ 8.821612] pstore: using deflate compression [ 8.825726] Platform Keyring initialized [ 8.926529] NET: Registered protocol family 38 [ 8.927748] Key type asymmetric registered [ 8.928749] Asymmetric key parser 'x509' registered [ 8.930886] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 8.933810] io scheduler mq-deadline registered [ 8.935156] io scheduler kyber registered [ 8.936825] io scheduler bfq registered [ 8.938846] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 8.942298] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 8.944804] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 8.948342] ACPI: Power Button [PWRF] [ 8.954465] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 8.962639] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 8.979612] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 8.996490] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 9.023407] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 9.063804] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 9.104164] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 9.112469] Non-volatile memory driver v1.3 [ 9.113979] Linux agpgart interface v0.103 [ 9.173345] virtio_blk virtio1: [vda] 134096 512-byte logical blocks (68.7 MB/65.5 MiB) [ 9.179057] vda: detected capacity change from 0 to 68657152 [ 9.205435] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 9.213145] vdb: detected capacity change from 0 to 1073741824 [ 9.263350] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 9.267749] vdc: detected capacity change from 0 to 2621440000 [ 9.307238] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 9.311342] vdd: detected capacity change from 0 to 2621440000 [ 9.327155] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 9.330626] vde: detected capacity change from 0 to 4294967296 [ 9.347432] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 9.350636] vdf: detected capacity change from 0 to 4294967296 [ 9.360994] libphy: Fixed MDIO Bus: probed [ 9.367697] usbcore: registered new interface driver usbserial_generic [ 9.370910] usbserial: USB Serial support registered for generic [ 9.373588] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 9.378349] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 9.380204] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 9.383152] mousedev: PS/2 mouse device common for all mice [ 9.386325] rtc_cmos 00:05: RTC can wake from S4 [ 9.388064] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 9.392762] rtc_cmos 00:05: registered as rtc0 [ 9.400090] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 9.404534] intel_pstate: CPU model not supported [ 9.410990] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 9.422838] hid: raw HID events driver (C) Jiri Kosina [ 9.425858] usbcore: registered new interface driver usbhid [ 9.428624] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 9.486429] usbhid: USB HID core driver [ 9.486606] drop_monitor: Initializing network drop monitor service [ 9.489184] Initializing XFRM netlink socket [ 9.516129] NET: Registered protocol family 10 [ 9.524862] Segment Routing with IPv6 [ 9.530880] NET: Registered protocol family 17 [ 9.532775] mpls_gso: MPLS GSO support [ 9.546817] RAS: Correctable Errors collector initialized. [ 9.552878] AVX version of gcm_enc/dec engaged. [ 9.557893] AES CTR mode by8 optimization enabled [ 9.698200] sched_clock: Marking stable (9698055531, 0)->(13646138911, -3948083380) [ 9.703807] registered taskstats version 1 [ 9.706784] Loading compiled-in X.509 certificates [ 9.710055] zswap: loaded using pool lzo/zbud [ 9.745373] Key type big_key registered [ 10.037915] Key type encrypted registered [ 10.039593] ima: No TPM chip found, activating TPM-bypass! [ 10.042415] ima: Allocated hash algorithm: sha1 [ 10.045119] ima: No architecture policies found [ 10.046724] evm: Initialising EVM extended attributes: [ 10.048391] evm: security.selinux [ 10.049374] evm: security.ima [ 10.050348] evm: security.capability [ 10.051429] evm: HMAC attrs: 0x1 [ 10.053678] rtc_cmos 00:05: setting system clock to 2026-01-16 06:55:47 UTC (1768546547) [ 10.062898] debug: unmapping init [mem 0xffffffffb4203000-0xffffffffb43fffff] [ 10.066166] debug: unmapping init [mem 0xffffffffb2f82000-0xffffffffb3258fff] [ 10.070052] Write protecting the kernel read-only data: 28672k [ 10.073329] debug: unmapping init [mem 0xffffffffb1603000-0xffffffffb17fffff] [ 10.076278] debug: unmapping init [mem 0xffffffffb1f14000-0xffffffffb1ffffff] [ 10.132332] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 10.147554] systemd[1]: Detected virtualization kvm. [ 10.150637] systemd[1]: Detected architecture x86-64. [ 10.153573] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 10.189614] systemd[1]: No hostname configured. [ 10.191210] systemd[1]: Set hostname to . [ 10.193233] random: systemd: uninitialized urandom read (16 bytes read) [ 10.197783] systemd[1]: Initializing machine ID from random generator. [ 10.335093] random: ln: uninitialized urandom read (6 bytes read) [ 10.517602] random: systemd: uninitialized urandom read (16 bytes read) [ 10.524555] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 10.542931] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 10.557043] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Slices. [ OK ] Reached target Timers. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Swap. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket. [ OK ] Reached target Sockets. Starting Apply Kernel Variables... Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... Starting Journal Service... [ OK ] Started Memstrack Anylazing Service. Starting Create Volatile Files and Directories... [ OK ] Reached target Initrd Root Device. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 12.820455] device-mapper: uevent: version 1.0.3 [ 12.831839] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 14.861528] random: fast init done [ 14.906655] virtio_net virtio0 ens2: renamed from eth0 [ 16.972723] scsi host0: ata_piix [ 17.011259] scsi host1: ata_piix [ 17.012671] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 17.015249] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 21.611715] random: crng init done [ 21.614138] random: 7 urandom warning(s) missed due to ratelimiting [ 24.632989] dracut-initqueue[585]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 26.974554] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 31.912342] printk: systemd: 26 output lines suppressed due to ratelimiting [ 32.443508] SELinux: Disabled at runtime. [ 32.512851] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 32.523165] systemd[1]: Detected virtualization kvm. [ 32.524915] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 33.702945] systemd[1]: initrd-switch-root.service: Succeeded. [ 33.707305] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 33.720702] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 33.749109] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 33.850152] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 34.080606] systemd[1]: Starting Journal Service... Starting Journal Service... [ 34.126884] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target rpc_pipefs.target. [ OK ] Stopped target Switch Root. [ OK ] Listening on udev Kernel Socket. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-getty.slice. [ OK ] Listening on initctl Compatibility Named Pipe. Starting Remount Root and Kernel File Systems... Mounting Kernel Debug File System... Mounting Huge Pages File System... [ OK ] Created slice system-serial\x2dgetty.slice. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Created slice User and Session Slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting POSIX Message Queue File System... [ OK ] Reached target Slices. [ OK ] Reached target Paths. Starting Apply Kernel Variables... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ 35.782882] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... Starting Configure read-only root support... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 37.197041] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 38.802858] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 38.976047] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 39.373628] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 39.561499] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (9s / no limit) [** ] A start job is running for Configur…only root support (10s / no limit) [*** ] A start job is running for Configur…only root support (10s / no limit) [ *** ] A start job is running for Configur…only root support (11s / no limit)[ 45.276616] Key type dns_resolver registered [ *** ] A start job is running for Configur…only root support (11s / no limit) [ ***] A start job is running for Configur…only root support (12s / no limit)[ 46.192083] NFS: Registering the id_resolver key type [ 46.204157] Key type id_resolver registered [ 46.208826] Key type id_legacy registered [ **] A start job is running for Configur…only root support (12s / no limit) [ *] A start job is running for Configur…only root support (13s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Load/Save Random Seed. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started irqbalance daemon. Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started Login Service. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg219-server login: [ 90.485400] libcfs: loading out-of-tree module taints kernel. [ 90.543190] Key type ._llcrypt registered [ 90.545662] Key type .llcrypt registered [ 90.608662] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [ 101.385953] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 102.326810] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 102.341159] alg: No test for adler32 (adler32-zlib) [ 103.563570] Lustre: Lustre: Build Version: 2.17.50_30_gbb2e37c [ 104.114308] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 105.823173] Key type lgssc registered [ 106.798889] Lustre: Echo OBD driver; http://www.lustre.org/ [ 118.495377] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 144.091902] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 151.546536] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 151.568581] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 152.702353] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 152.721539] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 152.770446] Lustre: lustre-MDT0000: new disk, initializing [ 152.831366] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 152.844590] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 155.047708] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 164.266367] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 164.354516] Lustre: 6476:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 164.386310] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 164.389250] Lustre: Skipped 1 previous similar message [ 164.450815] Lustre: lustre-MDT0001: new disk, initializing [ 164.501585] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 164.528837] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 164.534874] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 166.898212] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 170.416457] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 175.894073] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 176.051329] Lustre: lustre-OST0000: new disk, initializing [ 176.054323] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 176.086461] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 179.745954] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 184.354769] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 184.364768] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 184.423225] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 189.533679] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 189.605254] Lustre: lustre-OST0001: new disk, initializing [ 189.608543] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 189.666340] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 189.884461] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 189.887393] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 189.903587] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 193.654941] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 201.698388] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 206.473632] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 215.114358] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing check_logdir /tmp/testlogs/ [ 218.840559] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing yml_node [ 222.097577] Lustre: DEBUG MARKER: Client: 2.17.50.30 [ 223.695302] Lustre: DEBUG MARKER: MDS: 2.17.50.30 [ 225.379536] Lustre: DEBUG MARKER: OSS: 2.17.50.30 [ 226.351713] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Jan 16 01:59:22 EST 2026 [ 237.497768] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 238.347485] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 251.360118] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 251.361930] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 251.369743] Lustre: Skipped 1 previous similar message [ 251.377718] Lustre: Skipped 3 previous similar messages [ 256.479712] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 256.488341] Lustre: Skipped 3 previous similar messages [ 257.302763] LustreError: 12167:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 257.368936] Lustre: server umount lustre-MDT0000 complete [ 259.810836] LustreError: 7406:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768546797 with bad export cookie 11390647429946455842 [ 259.813476] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 259.822810] LustreError: 7406:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 261.600130] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 261.606237] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 261.609837] Lustre: Skipped 3 previous similar messages [ 264.608401] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 264.613124] Lustre: Skipped 2 previous similar messages [ 269.731572] LustreError: 9883:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 269.731833] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 269.743939] LustreError: 9883:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 269.747607] Lustre: Skipped 1 previous similar message [ 274.399202] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 274.465252] LustreError: 12370:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 274.470751] LustreError: 12370:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 274.538659] Lustre: server umount lustre-MDT0001 complete [ 276.991330] LustreError: 12572:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 276.993759] LustreError: 12572:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 277.024685] Lustre: server umount lustre-OST0000 complete [ 279.584309] LustreError: 12772:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 279.587599] LustreError: 12772:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 279.687703] Lustre: server umount lustre-OST0001 complete [ 283.624461] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [ 288.285966] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 291.013080] hrtimer: interrupt took 4700875 ns [ 318.281420] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 318.484936] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 318.504512] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 318.573343] Lustre: lustre-MDT0000: new disk, initializing [ 318.623179] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 318.637219] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 321.689984] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 328.947096] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 329.011414] Lustre: 14941:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 329.069060] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 329.071315] Lustre: Skipped 1 previous similar message [ 329.116314] Lustre: lustre-MDT0001: new disk, initializing [ 329.206718] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 329.233508] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 329.240176] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 331.751781] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 336.175530] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 338.796201] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 342.642563] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 344.229560] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 344.296087] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 345.644719] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 350.270400] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 352.835498] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 352.958203] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 355.236636] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 355.361924] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 359.913131] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 359.918415] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 359.923967] LustreError: Skipped 1 previous similar message [ 359.924896] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 359.936672] Lustre: Skipped 1 previous similar message [ 362.676116] LustreError: 17886:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 362.678791] LustreError: 17886:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 362.728548] Lustre: server umount lustre-OST0000 complete [ 370.144426] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 370.145950] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 370.160151] Lustre: Skipped 1 previous similar message [ 371.296488] LustreError: 18087:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 371.299973] LustreError: 18087:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 371.555275] Lustre: server umount lustre-MDT0000 complete [ 374.641393] LustreError: 14935:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768546912 with bad export cookie 11390647429946457368 [ 374.649811] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 374.651529] LustreError: 14935:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 374.894456] Lustre: server umount lustre-MDT0001 complete [ 378.242563] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 02:01:54 (1768546914) [ 380.282882] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 381.587873] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 02:01:57 (1768546917) [ 387.308460] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 387.587744] LustreError: 18958:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 387.638926] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 387.648741] Lustre: Skipped 1 previous similar message [ 391.012856] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 392.674893] LustreError: 18958:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 397.526101] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 400.637213] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 404.137268] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 406.633758] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 411.203389] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 414.891672] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 417.898948] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 428.001646] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 428.010092] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 428.015577] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 428.021387] Lustre: Skipped 4 previous similar messages [ 432.120108] LustreError: 21196:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 432.122826] LustreError: 21196:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 432.212442] Lustre: server umount lustre-OST0000 complete [ 436.194916] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 436.203141] Lustre: Skipped 2 previous similar messages [ 440.723396] Lustre: server umount lustre-MDT0000 complete [ 441.313946] LustreError: 18957:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 441.322653] LustreError: 18957:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 442.991648] LustreError: 18938:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768546980 with bad export cookie 11390647429946458257 [ 442.994790] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 443.004867] LustreError: 18938:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 443.254556] Lustre: server umount lustre-MDT0001 complete [ 450.007582] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 451.672624] Key type lgssc unregistered [ 451.865321] LNet: 22132:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 451.877648] LNetError: 22132:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 451.889035] LNet: Removed LNI 192.168.202.119@tcp [ 452.455163] Key type .llcrypt unregistered [ 452.456368] Key type ._llcrypt unregistered [ 468.428118] Key type ._llcrypt registered [ 468.429597] Key type .llcrypt registered [ 468.514901] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 469.154680] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 469.247098] alg: No test for adler32 (adler32-zlib) [ 470.267588] Lustre: Lustre: Build Version: 2.17.50_30_gbb2e37c [ 470.513383] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 472.127768] Key type lgssc registered [ 472.979754] Lustre: Echo OBD driver; http://www.lustre.org/ [ 479.021935] loop: module loaded [ 479.066519] loop0: detected capacity change from 0 to 204800000 [ 479.193966] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 479.202271] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 479.387713] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 483.710843] loop0: detected capacity change from 0 to 204800000 [ 483.871312] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 483.876778] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 483.890558] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 487.966592] loop0: detected capacity change from 0 to 204800000 [ 491.961254] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 491.994832] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 493.236519] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 493.303082] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 493.368788] Lustre: lustre-MDT0000: new disk, initializing [ 493.433269] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 493.453696] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 496.213783] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 502.322800] loop1: detected capacity change from 0 to 204800000 [ 506.154584] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 506.294216] Lustre: lustre-OST07c6: new disk, initializing [ 506.301118] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 506.308470] Lustre: Skipped 1 previous similar message [ 506.344531] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 506.463070] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 506.468046] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 506.494754] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 509.805872] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 513.090134] Lustre: 24264:0:(osd_internal.h:1459:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 255 < left 264, rollback = 2 [ 513.094920] Lustre: 24264:0:(osd_handler.c:2096:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 513.097702] Lustre: 24264:0:(osd_handler.c:2103:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 513.100817] Lustre: 24264:0:(osd_handler.c:2113:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 513.103610] Lustre: 24264:0:(osd_handler.c:2120:osd_trans_dump_creds()) insert: 4/65/3, delete: 0/0/0 [ 513.106270] Lustre: 24264:0:(osd_handler.c:2127:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 530.442871] Lustre: 24263:0:(osd_internal.h:1459:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 530.451156] Lustre: 24263:0:(osd_handler.c:2096:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 530.456737] Lustre: 24263:0:(osd_handler.c:2103:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 530.461849] Lustre: 24263:0:(osd_handler.c:2113:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/4 [ 530.465711] Lustre: 24263:0:(osd_handler.c:2120:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 530.470739] Lustre: 24263:0:(osd_handler.c:2127:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 534.623591] Lustre: 24263:0:(osd_internal.h:1459:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 534.630734] Lustre: 24263:0:(osd_internal.h:1459:osd_trans_exec_op()) Skipped 1 previous similar message [ 534.636450] Lustre: 24263:0:(osd_handler.c:2096:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 534.639451] Lustre: 24263:0:(osd_handler.c:2096:osd_trans_dump_creds()) Skipped 1 previous similar message [ 534.642759] Lustre: 24263:0:(osd_handler.c:2103:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 534.646307] Lustre: 24263:0:(osd_handler.c:2103:osd_trans_dump_creds()) Skipped 1 previous similar message [ 534.649716] Lustre: 24263:0:(osd_handler.c:2113:osd_trans_dump_creds()) write: 2/23/0, punch: 0/0/0, quota 4/150/0 [ 534.653448] Lustre: 24263:0:(osd_handler.c:2113:osd_trans_dump_creds()) Skipped 1 previous similar message [ 534.656722] Lustre: 24263:0:(osd_handler.c:2120:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 534.659551] Lustre: 24263:0:(osd_handler.c:2120:osd_trans_dump_creds()) Skipped 1 previous similar message [ 534.664432] Lustre: 24263:0:(osd_handler.c:2127:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 534.667192] Lustre: 24263:0:(osd_handler.c:2127:osd_trans_dump_creds()) Skipped 1 previous similar message [ 537.785897] Lustre: 25926:0:(osd_internal.h:1459:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 537.792111] Lustre: 25926:0:(osd_internal.h:1459:osd_trans_exec_op()) Skipped 1 previous similar message [ 537.796163] Lustre: 25926:0:(osd_handler.c:2096:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 537.799920] Lustre: 25926:0:(osd_handler.c:2096:osd_trans_dump_creds()) Skipped 1 previous similar message [ 537.803574] Lustre: 25926:0:(osd_handler.c:2103:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 537.807812] Lustre: 25926:0:(osd_handler.c:2103:osd_trans_dump_creds()) Skipped 1 previous similar message [ 537.810599] Lustre: 25926:0:(osd_handler.c:2113:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/0 [ 537.813783] Lustre: 25926:0:(osd_handler.c:2113:osd_trans_dump_creds()) Skipped 1 previous similar message [ 537.816475] Lustre: 25926:0:(osd_handler.c:2120:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 537.819673] Lustre: 25926:0:(osd_handler.c:2120:osd_trans_dump_creds()) Skipped 1 previous similar message [ 537.823909] Lustre: 25926:0:(osd_handler.c:2127:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 537.827683] Lustre: 25926:0:(osd_handler.c:2127:osd_trans_dump_creds()) Skipped 1 previous similar message [ 541.866817] Lustre: 25926:0:(osd_internal.h:1459:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 258 < left 278, rollback = 2 [ 541.873741] Lustre: 25926:0:(osd_internal.h:1459:osd_trans_exec_op()) Skipped 1 previous similar message [ 541.875931] Lustre: 25926:0:(osd_handler.c:2096:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 541.878507] Lustre: 25926:0:(osd_handler.c:2096:osd_trans_dump_creds()) Skipped 1 previous similar message [ 541.881262] Lustre: 25926:0:(osd_handler.c:2103:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 541.883651] Lustre: 25926:0:(osd_handler.c:2103:osd_trans_dump_creds()) Skipped 1 previous similar message [ 541.885885] Lustre: 25926:0:(osd_handler.c:2113:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/0 [ 541.888685] Lustre: 25926:0:(osd_handler.c:2113:osd_trans_dump_creds()) Skipped 1 previous similar message [ 541.891244] Lustre: 25926:0:(osd_handler.c:2120:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 541.893809] Lustre: 25926:0:(osd_handler.c:2120:osd_trans_dump_creds()) Skipped 1 previous similar message [ 541.896390] Lustre: 25926:0:(osd_handler.c:2127:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 541.898803] Lustre: 25926:0:(osd_handler.c:2127:osd_trans_dump_creds()) Skipped 1 previous similar message [ 553.273096] Lustre: 24264:0:(osd_internal.h:1459:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 553.277895] Lustre: 24264:0:(osd_internal.h:1459:osd_trans_exec_op()) Skipped 5 previous similar messages [ 553.281182] Lustre: 24264:0:(osd_handler.c:2096:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 553.286924] Lustre: 24264:0:(osd_handler.c:2096:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 553.290964] Lustre: 24264:0:(osd_handler.c:2103:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 553.294499] Lustre: 24264:0:(osd_handler.c:2103:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 553.298756] Lustre: 24264:0:(osd_handler.c:2113:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/0 [ 553.303295] Lustre: 24264:0:(osd_handler.c:2113:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 553.307024] Lustre: 24264:0:(osd_handler.c:2120:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 553.310914] Lustre: 24264:0:(osd_handler.c:2120:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 553.315511] Lustre: 24264:0:(osd_handler.c:2127:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 553.318772] Lustre: 24264:0:(osd_handler.c:2127:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 598.497360] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 598.502562] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 598.509236] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 603.618809] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 604.422187] LustreError: 26444:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 604.491177] Lustre: server umount lustre-OST07c6 complete [ 610.807650] LustreError: 26945:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 610.812738] LustreError: 26945:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 610.954956] Lustre: server umount lustre-MDT0000 complete [ 621.291813] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 623.136735] Key type lgssc unregistered [ 623.307372] LNet: 27777:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 623.312131] LNetError: 27777:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 623.330064] LNet: Removed LNI 192.168.202.119@tcp [ 623.876084] Key type .llcrypt unregistered [ 623.877610] Key type ._llcrypt unregistered [ 635.479852] Key type ._llcrypt registered [ 635.483193] Key type .llcrypt registered [ 635.592483] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 636.322579] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 636.354820] alg: No test for adler32 (adler32-zlib) [ 637.302451] Lustre: Lustre: Build Version: 2.17.50_30_gbb2e37c [ 637.458975] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 639.095563] Key type lgssc registered [ 639.739714] Lustre: Echo OBD driver; http://www.lustre.org/ [ 645.372131] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 645.421353] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 646.742528] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 649.512904] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 654.656738] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 654.877823] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 657.318948] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 660.727715] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 662.763692] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 666.049358] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 666.226362] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 668.913183] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 672.075751] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 672.359515] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 691.171212] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 691.172028] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 691.181820] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 691.188025] Lustre: Skipped 1 previous similar message [ 694.901299] LustreError: 31476:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 694.958676] Lustre: server umount lustre-OST0000 complete [ 700.897796] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 700.912725] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 700.933782] Lustre: Skipped 1 previous similar message [ 700.941078] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 706.528641] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 706.533644] Lustre: Skipped 3 previous similar messages [ 710.626669] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 710.635483] Lustre: Skipped 1 previous similar message [ 711.135155] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 711.235496] LustreError: 31679:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 711.240849] LustreError: 31679:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 711.283514] Lustre: server umount lustre-MDT0000 complete [ 713.391790] LustreError: 30624:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768547250 with bad export cookie 3953242827016366662 [ 713.392060] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 713.396785] LustreError: 30624:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 713.477559] LustreError: 31881:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 713.480202] LustreError: 31881:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 713.583404] Lustre: server umount lustre-MDT0001 complete [ 719.790983] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 721.490789] Key type lgssc unregistered [ 721.720596] LNet: 32413:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 721.724371] LNetError: 32413:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 721.735294] LNet: Removed LNI 192.168.202.119@tcp [ 722.220621] Key type .llcrypt unregistered [ 722.222594] Key type ._llcrypt unregistered [ 732.500026] Key type ._llcrypt registered [ 732.501762] Key type .llcrypt registered [ 732.559667] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 733.124929] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 733.146435] alg: No test for adler32 (adler32-zlib) [ 734.078679] Lustre: Lustre: Build Version: 2.17.50_30_gbb2e37c [ 734.193931] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 735.807178] Key type lgssc registered [ 736.395141] Lustre: Echo OBD driver; http://www.lustre.org/ [ 741.217333] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 741.228076] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 742.516861] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 745.255766] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 751.088432] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 751.334197] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 754.271730] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 757.452953] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 759.657535] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 763.882685] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 764.164621] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 767.275367] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 768.006882] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 771.345023] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 786.917226] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 786.924036] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 786.939422] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 786.942518] Lustre: Skipped 1 previous similar message [ 792.034761] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 792.038768] Lustre: Skipped 1 previous similar message [ 797.151110] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 797.154981] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 797.159566] Lustre: Skipped 1 previous similar message [ 797.182243] LustreError: 35900:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 797.240197] Lustre: server umount lustre-OST0000 complete [ 802.273927] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 802.282450] Lustre: Skipped 1 previous similar message [ 802.287307] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 802.290483] Lustre: Skipped 1 previous similar message [ 805.269530] LustreError: 36102:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 805.272588] LustreError: 36102:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 805.334321] Lustre: server umount lustre-MDT0000 complete [ 807.394131] LustreError: 33640:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 807.405733] LustreError: 33640:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 807.622978] LustreError: 33623:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768547345 with bad export cookie 5152884772008420337 [ 807.626191] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 807.627482] LustreError: 33623:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 807.740830] LustreError: 36303:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 807.743668] LustreError: 36303:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 807.863653] Lustre: server umount lustre-MDT0001 complete [ 813.617691] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 815.375311] Key type lgssc unregistered [ 815.579459] LNet: 36835:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 815.582603] LNetError: 36835:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 815.591788] LNet: Removed LNI 192.168.202.119@tcp [ 816.115594] Key type .llcrypt unregistered [ 816.117726] Key type ._llcrypt unregistered [ 832.053881] Key type ._llcrypt registered [ 832.055479] Key type .llcrypt registered [ 832.216709] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 833.060130] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 833.074928] alg: No test for adler32 (adler32-zlib) [ 834.209696] Lustre: Lustre: Build Version: 2.17.50_30_gbb2e37c [ 834.547704] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 836.352122] Key type lgssc registered [ 839.147418] Lustre: Echo OBD driver; http://www.lustre.org/ [ 854.623377] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 854.637559] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 856.439683] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 863.339146] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 865.768031] LustreError: 38212:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 865.778751] LustreError: 38212:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 870.897789] LustreError: 38212:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 876.001496] LustreError: 38212:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 877.715795] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 878.092646] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 883.357727] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 894.743922] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 902.856650] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 916.714701] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 917.516832] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 923.875194] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 925.782739] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 931.519820] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 939.891615] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.19@tcp (stopping) [ 940.000884] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 940.008680] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 941.030288] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 941.038908] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 941.042938] Lustre: Skipped 2 previous similar messages [ 941.082294] Lustre: Skipped 1 previous similar message [ 944.979163] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.19@tcp (stopping) [ 944.985897] Lustre: Skipped 1 previous similar message [ 945.976326] LustreError: 40461:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 946.146645] LustreError: 38924:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 946.176421] LustreError: 38924:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 946.214211] Lustre: server umount lustre-MDT0000 complete [ 950.446409] LustreError: 38192:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768547487 with bad export cookie 4851886081946291908 [ 950.453471] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 950.460608] LustreError: 38192:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 951.278205] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 951.280960] LustreError: 38209:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 951.322442] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 951.355027] LustreError: 38209:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 956.696417] LustreError: 40663:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 956.701682] LustreError: 40663:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 956.961984] Lustre: server umount lustre-MDT0001 complete [ 978.384502] LustreError: 40888:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 978.389584] LustreError: 40888:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 978.589876] Lustre: server umount lustre-OST0000 complete [ 995.723181] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 999.072291] Key type lgssc unregistered [ 999.508064] LNet: 41726:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 999.513270] LNetError: 41726:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 999.536093] LNet: Removed LNI 192.168.202.119@tcp [ 1000.378324] Key type .llcrypt unregistered [ 1000.381986] Key type ._llcrypt unregistered [ 1026.566884] Key type ._llcrypt registered [ 1026.568941] Key type .llcrypt registered [ 1026.675658] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 1027.997413] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1028.034053] alg: No test for adler32 (adler32-zlib) [ 1029.226035] Lustre: Lustre: Build Version: 2.17.50_30_gbb2e37c [ 1029.476660] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 1031.151186] Key type lgssc registered [ 1032.831739] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1041.641142] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1041.652275] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1043.221218] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1048.862071] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 1052.706178] LustreError: 43125:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1052.750184] LustreError: 43125:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1057.764745] LustreError: 43125:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1059.597135] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1060.171221] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1066.462660] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 1074.053758] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1078.840970] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1086.583921] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1086.990250] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1089.015937] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1093.945697] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 1099.237495] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1109.490827] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1109.499912] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1109.536262] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1110.498980] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1110.506211] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1111.505416] LustreError: 45369:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1111.673358] Lustre: server umount lustre-OST0000 complete [ 1132.001085] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1132.012741] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1132.033911] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1132.042842] Lustre: Skipped 1 previous similar message [ 1136.096676] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1136.104840] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1137.998221] LustreError: 45745:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1138.013295] LustreError: 45745:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1138.221631] Lustre: server umount lustre-MDT0000 complete [ 1141.226248] LustreError: 44019:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1141.254544] LustreError: 44019:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1142.980066] LustreError: 43106:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768547680 with bad export cookie 7328066951716010444 [ 1142.988797] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1142.990024] LustreError: 43106:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1149.276801] LustreError: 45949:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1149.290160] LustreError: 45949:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1149.589665] Lustre: server umount lustre-MDT0001 complete [ 1164.129839] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 1167.341918] Key type lgssc unregistered [ 1167.662948] LNet: 46482:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1167.677770] LNetError: 46482:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1167.731320] LNet: Removed LNI 192.168.202.119@tcp [ 1169.030991] Key type .llcrypt unregistered [ 1169.032760] Key type ._llcrypt unregistered [ 1195.496255] Key type ._llcrypt registered [ 1195.501804] Key type .llcrypt registered [ 1195.680828] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 1197.371627] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1197.492690] alg: No test for adler32 (adler32-zlib) [ 1198.812456] Lustre: Lustre: Build Version: 2.17.50_30_gbb2e37c [ 1199.187152] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 1200.959217] Key type lgssc registered [ 1202.665244] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1211.448063] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1211.469834] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1213.114445] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1218.874397] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 1222.628421] LustreError: 47879:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1222.651594] LustreError: 47879:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1227.754542] LustreError: 47879:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1229.867924] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1230.130596] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1235.046986] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 1241.923844] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1246.485687] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1255.477435] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1255.756060] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1257.840992] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1262.887818] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 1271.197792] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1276.483338] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1280.371669] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1293.338648] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-01-16 2h17m07s [ 1295.616346] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1298.803911] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.19@tcp (stopping) [ 1298.912766] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1298.925523] Lustre: Skipped 2 previous similar messages [ 1299.863660] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.19@tcp (stopping) [ 1299.880930] Lustre: Skipped 3 previous similar messages [ 1304.306618] LustreError: 50443:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1304.434494] Lustre: server umount lustre-MDT0000 complete [ 1304.933691] LustreError: 47875:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.19@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1308.887558] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1311.067736] LustreError: 47875:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.19@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1311.092887] LustreError: 47875:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1317.202901] LustreError: 47876:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.19@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1317.222328] LustreError: 47876:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1317.353580] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1317.521645] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1317.825416] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1317.907265] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1318.845606] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1323.503899] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 1328.066808] Lustre: DEBUG MARKER: Wait for df (36047) ... [ 1329.913194] Lustre: DEBUG MARKER: done [ 1338.337422] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1338.344241] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1338.357520] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1338.367796] Lustre: Skipped 3 previous similar messages [ 1338.857203] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1344.520275] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1344.533694] Lustre: Skipped 3 previous similar messages [ 1349.104532] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1349.135098] LustreError: 51497:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1349.138087] LustreError: 51497:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1349.233211] Lustre: server umount lustre-OST0000 complete [ 1354.727300] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1354.734103] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1354.762760] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1354.789251] Lustre: Skipped 2 previous similar messages [ 1354.820612] Lustre: Skipped 2 previous similar messages [ 1360.123954] LustreError: 51700:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1360.130142] LustreError: 51700:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1360.523367] Lustre: server umount lustre-MDT0000 complete [ 1362.405433] LustreError: 47874:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1362.429208] LustreError: 47874:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1365.275462] LustreError: 47861:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768547902 with bad export cookie 3716389525979405881 [ 1365.279850] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1365.289239] LustreError: 47861:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1365.426817] LustreError: 51903:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1365.432014] LustreError: 51903:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1365.617670] Lustre: server umount lustre-MDT0001 complete [ 1380.901534] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 1384.305768] Key type lgssc unregistered [ 1384.608987] LNet: 52435:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1384.616779] LNetError: 52435:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1385.640481] LNet: Removed LNI 192.168.202.119@tcp [ 1386.677360] Key type .llcrypt unregistered [ 1386.679991] Key type ._llcrypt unregistered [ 1438.421898] Key type ._llcrypt registered [ 1438.424965] Key type .llcrypt registered [ 1438.607823] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 1440.502076] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1440.578425] alg: No test for adler32 (adler32-zlib) [ 1441.743712] Lustre: Lustre: Build Version: 2.17.50_30_gbb2e37c [ 1441.984980] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 1443.727867] Key type lgssc registered [ 1445.175537] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1455.073239] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1455.096184] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1456.519791] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1456.547472] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1456.795498] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1462.747459] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 1473.573814] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1473.788503] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1473.870908] Lustre: 54624:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1474.420362] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1480.123675] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 1488.003922] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1492.490869] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1500.577487] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1500.876572] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1501.178182] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1508.734720] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 1509.572727] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1516.580200] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1524.205823] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-01-16 2h20m59s [ 1526.422073] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1534.942730] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1538.650278] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1538.725405] LustreError: 54630:0:(mdt_handler.c:2334:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1558.944259] Lustre: lustre-MDT0000: Client 055554f7-369c-42ff-8f55-0e679b254653 (at 192.168.202.19@tcp) reconnecting [ 1578.799831] LustreError: 54630:0:(mdt_handler.c:2334:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1582.629532] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1586.781838] Lustre: DEBUG MARKER: done [ 1596.399191] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1596.414726] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1596.433424] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1601.216361] LustreError: 57665:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1601.652533] Lustre: server umount lustre-OST0000 complete [ 1607.142608] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1607.160219] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1607.162746] Lustre: Skipped 2 previous similar messages [ 1607.190223] Lustre: Skipped 2 previous similar messages [ 1608.229779] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1608.242615] Lustre: Skipped 1 previous similar message [ 1612.588844] LustreError: 57866:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1612.594458] LustreError: 57866:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1612.780143] Lustre: server umount lustre-MDT0000 complete [ 1617.516656] LustreError: 54618:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768548154 with bad export cookie 17889907527114441014 [ 1617.530450] LustreError: 54618:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1617.530981] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1617.761472] LustreError: 58068:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1617.775476] LustreError: 58068:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1617.939188] Lustre: server umount lustre-MDT0001 complete [ 1632.248909] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 1635.761603] Key type lgssc unregistered [ 1636.127842] LNet: 58603:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1636.134160] LNetError: 58603:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1636.162597] LNet: Removed LNI 192.168.202.119@tcp [ 1637.295891] Key type .llcrypt unregistered [ 1637.302841] Key type ._llcrypt unregistered [ 1696.595428] Key type ._llcrypt registered [ 1696.599646] Key type .llcrypt registered [ 1696.740758] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 1698.377784] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1698.400613] alg: No test for adler32 (adler32-zlib) [ 1699.694453] Lustre: Lustre: Build Version: 2.17.50_30_gbb2e37c [ 1700.046316] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 1701.752655] Key type lgssc registered [ 1703.179889] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1707.138552] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1707.173289] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1708.592138] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1708.635228] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1708.847723] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1710.121758] Lustre: Failing over lustre-MDT0000 [ 1710.359479] LustreError: 60831:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1710.562731] Lustre: server umount lustre-MDT0000 complete [ 1718.556279] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 02:24:13 (1768548253) [ 1726.863107] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1727.687040] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1733.857182] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 1745.868718] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1746.132980] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1746.193496] Lustre: 61386:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1746.424859] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 1746.764792] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1753.825579] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 1762.220673] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1767.119349] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1776.772871] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1776.946860] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1777.181919] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1782.984883] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1785.123136] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 1791.966285] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1799.420654] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1808.352987] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1808.378474] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1808.404461] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1813.840914] LustreError: 63759:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1813.856831] LustreError: 63759:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1814.145178] Lustre: server umount lustre-MDT0000 complete [ 1818.567515] LustreError: 61379:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768548356 with bad export cookie 754402580276043399 [ 1818.575173] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1818.745734] LustreError: 63961:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1818.762851] LustreError: 63961:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1819.105455] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1819.108525] LustreError: 63573:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1819.114240] Lustre: Skipped 2 previous similar messages [ 1819.135336] LustreError: 63573:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 1819.233679] Lustre: server umount lustre-MDT0001 complete [ 1823.848243] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1828.786337] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1838.700699] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1843.620202] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xa782d2656fde76a [ 1843.635039] Lustre: MGC192.168.202.119@tcp: Connection restored to 0@lo (at 0@lo) [ 1844.163763] LustreError: 63011:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1844.316326] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1844.471312] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1848.420514] LustreError: 63011:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1848.429986] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1849.843222] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 1853.944536] LustreError: 63011:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1858.026549] LustreError: 64817:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1858.069440] LustreError: 64817:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1863.338902] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1863.788317] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1868.790659] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1872.356561] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 1879.824033] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1884.958895] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1894.372770] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1894.374706] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1894.375705] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1894.375710] Lustre: Skipped 2 previous similar messages [ 1894.427020] Lustre: Skipped 1 previous similar message [ 1898.647333] LustreError: 66220:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1898.651371] LustreError: 66220:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1898.698851] Lustre: server umount lustre-MDT0000 complete [ 1899.491088] LustreError: 64819:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1899.523012] LustreError: 64819:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 1903.970134] LustreError: 66221:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768548441 with bad export cookie 754402580276045674 [ 1903.971954] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1903.986576] LustreError: 66221:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1904.125441] LustreError: 66423:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1904.136884] LustreError: 66423:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1904.351311] Lustre: server umount lustre-MDT0001 complete [ 1911.892029] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1920.996049] Lustre: 60309:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1768548442/real 1768548442] req@00000000c506e5e9 x1854457236230528/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1768548458 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1921.034883] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1921.045023] Lustre: Skipped 1 previous similar message [ 1923.801868] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1925.159719] Lustre: 60309:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1768548446/real 1768548446] req@000000000e631ca6 x1854457236230656/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1768548462 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1929.639085] LustreError: 65687:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1929.744501] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1929.898659] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 1930.193320] Lustre: 60312:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1768548451/real 1768548451] req@0000000065f04bd5 x1854457236231040/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1768548467 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1930.244449] Lustre: 60312:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1934.751195] Lustre: 60310:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1768548456/real 1768548456] req@00000000760ada4c x1854457236231168/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1768548472 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1935.817495] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 1945.728669] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1950.126762] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1950.996788] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 1957.219996] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1961.902739] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1969.262241] LustreError: 68676:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1969.278586] LustreError: 68676:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1969.626869] Lustre: server umount lustre-MDT0000 complete [ 1970.667652] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1970.680611] LustreError: 63011:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1970.699422] Lustre: Skipped 1 previous similar message [ 1970.727723] LustreError: 63011:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 1974.035840] LustreError: 61379:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768548511 with bad export cookie 754402580276047046 [ 1974.043474] LustreError: 61379:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1974.044269] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1974.413883] Lustre: server umount lustre-MDT0001 complete [ 1981.709434] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1985.883352] Lustre: DEBUG MARKER: files compared the same [ 1998.237486] LustreError: 69472:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1998.250932] LustreError: 69472:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 1998.574290] Lustre: server umount lustre-OST0000 complete [ 2021.340906] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 2025.203390] Key type lgssc unregistered [ 2025.835560] LNet: 70311:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2025.845195] LNetError: 70311:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2025.924028] LNet: Removed LNI 192.168.202.119@tcp [ 2027.162466] Key type .llcrypt unregistered [ 2027.166957] Key type ._llcrypt unregistered [ 2056.638465] Key type ._llcrypt registered [ 2056.641453] Key type .llcrypt registered [ 2056.708119] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 2058.232142] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2058.264285] alg: No test for adler32 (adler32-zlib) [ 2059.311125] Lustre: Lustre: Build Version: 2.17.50_30_gbb2e37c [ 2059.543881] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 2061.239346] Key type lgssc registered [ 2062.431486] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2071.892674] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2071.917424] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2073.459028] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2078.439460] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug +malloc all [ 2082.805401] LustreError: 71709:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2082.843556] LustreError: 71709:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2087.908312] LustreError: 71709:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2088.962468] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2089.333177] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2094.230865] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug +malloc all [ 2101.540133] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2106.893607] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2114.894878] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2115.307889] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2118.383943] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2122.388809] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug +malloc all [ 2128.817880] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2144.227962] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2144.228787] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2144.231994] LustreError: Skipped 1 previous similar message [ 2144.251337] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2144.265144] Lustre: Skipped 1 previous similar message [ 2144.287240] Lustre: Skipped 1 previous similar message [ 2148.480858] LustreError: 73971:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2148.869917] Lustre: server umount lustre-OST0000 complete [ 2156.001499] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2156.006056] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2156.025432] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2159.583669] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2159.606337] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2160.784853] LustreError: 74175:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2160.799399] LustreError: 74175:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2161.191651] Lustre: server umount lustre-MDT0000 complete [ 2163.748113] LustreError: 71709:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2163.773597] LustreError: 71709:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2166.080689] LustreError: 71689:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768548703 with bad export cookie 12570715809501645918 [ 2166.082461] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2166.088166] LustreError: 71689:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2166.263606] LustreError: 74376:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2166.271836] LustreError: 74376:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2166.736732] Lustre: server umount lustre-MDT0001 complete [ 2179.987443] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 2183.741846] Key type lgssc unregistered [ 2184.081434] LNet: 74911:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2184.123158] LNetError: 74911:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2184.139252] LNet: Removed LNI 192.168.202.119@tcp [ 2185.192767] Key type .llcrypt unregistered [ 2185.195385] Key type ._llcrypt unregistered [ 2211.778630] Key type ._llcrypt registered [ 2211.782749] Key type .llcrypt registered [ 2211.939510] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 2213.360173] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2213.482926] alg: No test for adler32 (adler32-zlib) [ 2214.835263] Lustre: Lustre: Build Version: 2.17.50_30_gbb2e37c [ 2215.165075] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 2216.815396] Key type lgssc registered [ 2218.496800] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2227.846611] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2227.871610] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2243.424322] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2248.543220] LustreError: 76279:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.119@tcp: failed processing log, type 4: rc = -110 [ 2275.295235] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2282.600469] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 2288.614968] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2297.470154] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2297.632454] Lustre: *** cfs_fail_loc=706, val=0*** [ 2298.113992] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2298.221214] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2302.539781] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 2303.460123] LustreError: 76305:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2303.493208] LustreError: 76305:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2308.584782] LustreError: 76305:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2309.614306] LustreError: 77250:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2309.635741] LustreError: 77250:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2312.832453] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2313.329132] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2319.334343] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 2325.429579] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2329.556844] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2333.664109] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2333.684233] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2333.705542] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2337.987256] LustreError: 78602:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 2338.178444] Lustre: server umount lustre-OST0000 complete [ 2343.908741] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2343.924861] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2343.941884] Lustre: Skipped 1 previous similar message [ 2343.947185] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2343.953367] Lustre: Skipped 2 previous similar messages [ 2348.907049] LustreError: 78803:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 2348.914048] LustreError: 78803:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2349.028108] Lustre: server umount lustre-MDT0000 complete [ 2349.542774] LustreError: 77255:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2349.564966] LustreError: 77255:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2352.620463] LustreError: 76287:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768548890 with bad export cookie 15901786302839625168 [ 2352.625945] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2352.629807] LustreError: 76287:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2352.765391] LustreError: 79007:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2352.772480] LustreError: 79007:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2353.020607] Lustre: server umount lustre-MDT0001 complete [ 2364.210495] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 2367.581745] Key type lgssc unregistered [ 2367.860651] LNet: 79540:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2367.871613] LNetError: 79540:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2367.887814] LNet: Removed LNI 192.168.202.119@tcp [ 2368.793350] Key type .llcrypt unregistered [ 2368.794976] Key type ._llcrypt unregistered [ 2399.047757] Key type ._llcrypt registered [ 2399.049616] Key type .llcrypt registered [ 2399.181185] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 2400.196265] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2400.399167] alg: No test for adler32 (adler32-zlib) [ 2401.726492] Lustre: Lustre: Build Version: 2.17.50_30_gbb2e37c [ 2401.992965] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 2403.719207] Key type lgssc registered [ 2405.559875] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2416.318766] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2416.338617] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2422.881255] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 2432.695474] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2433.480818] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2438.254390] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 2438.636245] LustreError: 81608:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2438.667297] LustreError: 81608:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2442.720138] LustreError: 81613:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2442.767627] LustreError: 81613:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2447.843885] LustreError: 81609:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2447.860592] LustreError: 81609:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2448.994159] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2449.485647] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2454.513711] LustreError: 81608:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2454.547019] LustreError: 81608:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 2456.765437] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 2462.698493] LustreError: 81612:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2462.719681] LustreError: 81612:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2465.840663] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2465.848930] Lustre: MGS: Client 34632f77-e490-4e5e-a1e2-1a88921790a6 (at 0@lo) reconnecting [ 2466.115126] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2466.125945] LustreError: 81623:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2466.212636] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2470.887479] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 2486.759699] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2486.770824] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2486.783118] Lustre: Skipped 1 previous similar message [ 2486.789524] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2486.814437] Lustre: Skipped 1 previous similar message [ 2491.886813] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2491.896996] Lustre: Skipped 1 previous similar message [ 2492.167448] LustreError: 83591:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 2492.528377] Lustre: server umount lustre-OST0000 complete [ 2496.995638] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2497.024450] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2502.115302] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2502.132713] Lustre: Skipped 2 previous similar messages [ 2502.861824] LustreError: 83794:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2502.869536] LustreError: 83794:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2502.904129] Lustre: server umount lustre-MDT0000 complete [ 2507.235435] LustreError: 81607:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2507.257242] LustreError: 81607:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2507.550701] LustreError: 83995:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2507.564231] LustreError: 83995:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2507.999598] Lustre: server umount lustre-MDT0001 complete [ 2512.282595] LustreError: 84197:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2512.286420] LustreError: 84197:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2512.324935] Lustre: server umount MGS complete [ 2525.790726] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 02:37:41 (1768549061) [ 2557.956213] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [ 2569.415460] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 2608.701501] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2614.317216] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 2625.053898] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2625.401400] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2625.468333] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2625.568170] Lustre: lustre-MDT0001: new disk, initializing [ 2625.798451] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2625.872167] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2625.882799] Lustre: 88703:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2625.901788] LustreError: 88703:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2630.605686] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 2641.763994] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2642.039887] Lustre: lustre-OST0000: new disk, initializing [ 2642.049895] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2642.143266] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2648.745920] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 2652.768536] Lustre: 89893:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@00000000d94a8aaa x1854457972302592/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1768549206 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2657.963698] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2666.113774] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2666.131572] Lustre: MGS: Client b8ef1d72-e5e5-41b8-b6e1-1ce94914ebeb (at 0@lo) reconnecting [ 2666.217267] Lustre: 87987:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2666.255769] LustreError: 87987:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x0 : rc = -11 [ 2666.320452] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2666.484752] Lustre: lustre-MDT0000: new disk, initializing [ 2666.749759] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2666.812428] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2666.973728] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2667.875946] Lustre: 90412:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@00000000ae01e5b8 x1854457972308992/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1768549221 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2672.963845] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2672.976601] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2673.095912] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2673.111053] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 2703.840865] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2703.850866] Lustre: Skipped 1 previous similar message [ 2703.858655] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2703.864403] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2706.055298] LustreError: 90931:0:(obd_class.h:479:obd_check_dev()) Device 11 not setup [ 2706.065564] LustreError: 90931:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 2706.609628] Lustre: server umount lustre-OST0000 complete [ 2714.084599] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2714.088119] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2714.094724] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2714.094738] Lustre: Skipped 1 previous similar message [ 2714.134588] Lustre: Skipped 2 previous similar messages [ 2725.343182] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2725.464743] LustreError: 91133:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 2725.477643] LustreError: 91133:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2725.535216] Lustre: server umount lustre-MDT0000 complete [ 2729.442483] LustreError: 88673:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2729.477575] LustreError: 88673:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2736.865151] Lustre: server umount lustre-MDT0001 complete [ 2741.046725] Lustre: server umount MGS complete [ 2754.603512] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 02:41:30 (1768549290) [ 2773.859049] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 2777.912747] Key type lgssc unregistered [ 2778.395150] LNet: 92875:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2778.425086] LNetError: 92875:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2778.453837] LNet: Removed LNI 192.168.202.119@tcp [ 2780.005537] Key type .llcrypt unregistered [ 2780.013566] Key type ._llcrypt unregistered [ 2798.579624] Key type ._llcrypt registered [ 2798.589274] Key type .llcrypt registered [ 2798.803776] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 2800.585849] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2800.902871] alg: No test for adler32 (adler32-zlib) [ 2802.067362] Lustre: Lustre: Build Version: 2.17.50_30_gbb2e37c [ 2802.332102] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 2804.055234] Key type lgssc registered [ 2805.501801] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2814.835643] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2814.849946] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2816.362744] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2821.572905] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 2825.708542] LustreError: 94079:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2825.734863] LustreError: 94079:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2830.820167] LustreError: 94079:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2832.839123] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2833.483813] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2839.166893] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 2846.446301] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2851.309744] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2859.654523] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2860.129665] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2862.225499] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 2866.911497] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 2873.690249] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2888.161159] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2888.165038] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2888.179702] LustreError: Skipped 1 previous similar message [ 2888.180920] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2888.219160] Lustre: Skipped 1 previous similar message [ 2892.867270] LustreError: 96343:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2892.972646] Lustre: server umount lustre-OST0000 complete [ 2898.412986] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2898.417399] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2898.435318] Lustre: Skipped 1 previous similar message [ 2898.448348] Lustre: Skipped 2 previous similar messages [ 2903.522368] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2907.618484] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2907.623371] Lustre: Skipped 1 previous similar message [ 2911.201181] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2911.323466] LustreError: 96545:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2911.338906] LustreError: 96545:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2911.620852] Lustre: server umount lustre-MDT0000 complete [ 2912.745198] LustreError: 94080:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2912.780314] LustreError: 94080:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2916.154158] LustreError: 94060:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768549453 with bad export cookie 8298172220952520559 [ 2916.158964] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2916.170495] LustreError: 94060:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2916.347115] LustreError: 96750:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 2916.355807] LustreError: 96750:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2916.576574] Lustre: server umount lustre-MDT0001 complete [ 2929.942395] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 2933.067193] Key type lgssc unregistered [ 2933.282213] LNet: 97284:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2933.288260] LNetError: 97284:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2933.305991] LNet: Removed LNI 192.168.202.119@tcp [ 2933.916539] Key type .llcrypt unregistered [ 2933.917955] Key type ._llcrypt unregistered [ 2951.343229] Key type ._llcrypt registered [ 2951.344617] Key type .llcrypt registered [ 2951.442273] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 2952.462930] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2952.706556] alg: No test for adler32 (adler32-zlib) [ 2953.760646] Lustre: Lustre: Build Version: 2.17.50_30_gbb2e37c [ 2954.093722] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 2955.823326] Key type lgssc registered [ 2957.761980] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2966.461788] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2966.476505] LustreError: 98338:0:(tgt_mount.c:2439:server_fill_super()) cfs_race id 716 sleeping [ 2966.545620] LustreError: 98348:0:(tgt_mount.c:2439:server_fill_super()) cfs_fail_race id 716 waking [ 2966.553019] LustreError: 98338:0:(tgt_mount.c:2439:server_fill_super()) cfs_fail_race id 716 awake: rc=4935 [ 2966.555816] LustreError: 98338:0:(tgt_mount.c:2451:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 2966.571668] LustreError: 98338:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2966.614830] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2968.397761] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2977.792490] LustreError: 98385:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2977.822263] LustreError: 98385:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2978.940213] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2979.301553] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2985.116555] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 2992.358929] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing lsmod [ 2997.064483] LustreError: 99517:0:(tgt_mount.c:2439:server_fill_super()) cfs_race id 716 sleeping [ 2997.112732] LustreError: 99520:0:(tgt_mount.c:2439:server_fill_super()) cfs_fail_race id 716 waking [ 2997.118429] LustreError: 99517:0:(tgt_mount.c:2439:server_fill_super()) cfs_fail_race id 716 awake: rc=4950 [ 2997.128428] LustreError: 99517:0:(tgt_mount.c:2451:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 2997.146870] LustreError: 99517:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2997.155704] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2997.645761] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3001.197678] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3001.206253] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3001.213434] Lustre: Skipped 1 previous similar message [ 3005.350985] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3005.359030] Lustre: Skipped 4 previous similar messages [ 3007.449529] LustreError: 99705:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3007.810535] Lustre: server umount lustre-MDT0000 complete [ 3010.535047] LustreError: 98385:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3010.550172] LustreError: 98385:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3012.350356] LustreError: 98365:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768549549 with bad export cookie 11316354475450030478 [ 3012.353185] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3012.364671] LustreError: 98365:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3013.599281] LustreError: 98817:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000cb7d6a2a x1854458551096448/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3013.631219] LustreError: 99909:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 3013.647460] LustreError: 99909:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3013.871587] Lustre: server umount lustre-MDT0001 complete [ 3028.028218] LustreError: 100110:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 3028.034597] LustreError: 100110:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3028.166052] Lustre: server umount lustre-OST0000 complete [ 3037.879331] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3038.373505] LustreError: 100588:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3038.439633] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3043.809240] LustreError: 100588:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3044.282634] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 3047.917720] LustreError: 100588:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3054.938750] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3055.335349] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3061.107974] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 3068.874227] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3074.312787] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3084.150079] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3084.430113] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3090.938962] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 3100.165737] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3108.032480] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3120.613498] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3120.619748] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3120.632655] Lustre: Skipped 1 previous similar message [ 3120.639608] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3121.637340] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3125.008205] LustreError: 102855:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3125.011725] LustreError: 102855:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3125.305517] Lustre: server umount lustre-OST0000 complete [ 3131.875810] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3131.895658] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3131.905640] Lustre: Skipped 2 previous similar messages [ 3136.251563] LustreError: 103058:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3136.258521] LustreError: 103058:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3136.388122] Lustre: server umount lustre-MDT0000 complete [ 3136.997107] LustreError: 100582:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3137.014495] LustreError: 100582:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3140.807385] LustreError: 101306:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768549678 with bad export cookie 11316354475450031178 [ 3140.824835] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3141.245281] Lustre: server umount lustre-MDT0001 complete [ 3152.723284] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 3155.529173] Key type lgssc unregistered [ 3155.777270] LNet: 103794:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3155.782353] LNetError: 103794:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3155.799415] LNet: Removed LNI 192.168.202.119@tcp [ 3156.695208] Key type .llcrypt unregistered [ 3156.697980] Key type ._llcrypt unregistered [ 3183.913741] Key type ._llcrypt registered [ 3183.916705] Key type .llcrypt registered [ 3184.058581] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 3185.697995] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3185.991734] alg: No test for adler32 (adler32-zlib) [ 3187.248360] Lustre: Lustre: Build Version: 2.17.50_30_gbb2e37c [ 3187.490589] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 3189.191229] Key type lgssc registered [ 3190.807939] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3200.597535] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3200.654436] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3202.532875] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3208.072300] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 3211.781415] LustreError: 105194:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3211.810062] LustreError: 105194:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3216.876898] LustreError: 105194:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3219.906659] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3220.487559] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3226.398883] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 3233.854251] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3238.612584] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3247.050691] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3247.588398] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3255.405383] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 3262.461569] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3272.187330] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 3279.384368] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3283.669676] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3290.592228] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3290.605733] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3290.618106] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3292.666314] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3292.669161] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3292.669851] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3292.712247] Lustre: Skipped 1 previous similar message [ 3294.390320] LustreError: 107576:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3294.698259] Lustre: server umount lustre-OST0000 complete [ 3302.371614] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3302.388726] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3302.410898] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3304.924900] LustreError: 107778:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3304.931188] LustreError: 107778:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3305.218395] Lustre: server umount lustre-MDT0000 complete [ 3308.000894] LustreError: 105194:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3308.016206] LustreError: 105194:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3309.666938] LustreError: 105174:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768549847 with bad export cookie 17608585459595425655 [ 3309.677505] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3309.678081] LustreError: 105174:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3309.921441] LustreError: 107980:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 3309.934398] LustreError: 107980:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3310.332155] Lustre: server umount lustre-MDT0001 complete [ 3324.518287] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 3327.257745] Key type lgssc unregistered [ 3327.577767] LNet: 108514:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3327.583396] LNetError: 108514:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3328.613194] LNet: Removed LNI 192.168.202.119@tcp [ 3329.632920] Key type .llcrypt unregistered [ 3329.635883] Key type ._llcrypt unregistered [ 3348.327957] Key type ._llcrypt registered [ 3348.335273] Key type .llcrypt registered [ 3348.564988] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 3349.568214] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3349.605342] alg: No test for adler32 (adler32-zlib) [ 3350.777253] Lustre: Lustre: Build Version: 2.17.50_30_gbb2e37c [ 3351.011831] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 3352.711292] Key type lgssc registered [ 3354.044810] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3363.892385] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3363.930239] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3365.565372] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3370.853800] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 3375.083428] LustreError: 109717:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3375.110667] LustreError: 109717:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3380.218906] LustreError: 109717:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3382.530120] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3382.919038] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3389.335314] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 3397.941308] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3402.740123] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3411.755685] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3412.498403] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3421.849961] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 3429.643808] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3435.061322] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3448.069490] LustreError: 111981:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3448.359929] Lustre: server umount lustre-OST0000 complete [ 3454.432397] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3454.438300] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3454.453642] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3455.459712] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3455.461985] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3460.578533] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3460.591773] Lustre: Skipped 2 previous similar messages [ 3462.634498] LustreError: 112182:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3462.637306] LustreError: 112182:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3462.883322] Lustre: server umount lustre-MDT0000 complete [ 3463.584834] Lustre: 109027:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1768549985/real 1768549985] req@0000000084aaa0d2 x1854458967390976/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1768550001 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3463.629503] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3463.656524] LustreError: 109717:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3463.692448] LustreError: 109717:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3465.701187] Lustre: 109028:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1768549987/real 1768549987] req@00000000034c16bb x1854458967391872/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1768550003 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3468.082290] LustreError: 109697:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768550005 with bad export cookie 7466706767566343628 [ 3468.086927] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3468.099697] LustreError: 109697:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3468.249560] LustreError: 112384:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 3468.260580] LustreError: 112384:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3468.509469] Lustre: server umount lustre-MDT0001 complete [ 3484.569901] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 3488.369861] Key type lgssc unregistered [ 3488.885788] LNet: 112918:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3488.904218] LNetError: 112918:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3488.943208] LNet: Removed LNI 192.168.202.119@tcp [ 3490.018154] Key type .llcrypt unregistered [ 3490.023084] Key type ._llcrypt unregistered [ 3523.807542] Key type ._llcrypt registered [ 3523.810521] Key type .llcrypt registered [ 3524.039481] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 3525.911116] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3525.953346] alg: No test for adler32 (adler32-zlib) [ 3527.152552] Lustre: Lustre: Build Version: 2.17.50_30_gbb2e37c [ 3527.542363] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 3529.311441] Key type lgssc registered [ 3530.970681] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3541.582813] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3541.638596] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3543.415741] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3549.050214] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 3552.747728] LustreError: 114318:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3552.785059] LustreError: 114318:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3557.861643] LustreError: 114318:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3560.198201] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3561.084109] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3567.841944] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 3575.296352] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3581.653139] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3589.470922] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3589.969305] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3597.389324] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 3604.370372] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3613.572792] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3623.496173] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3629.067947] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3629.082962] Lustre: Skipped 2 previous similar messages [ 3629.373743] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3629.445228] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3634.993415] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3634.999547] Lustre: Skipped 2 previous similar messages [ 3638.028545] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3638.046245] Lustre: Skipped 1 previous similar message [ 3643.680818] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.202.19@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3643.693855] Lustre: Skipped 2 previous similar messages [ 3646.137530] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.202.19@tcp 192.168.0.[2,10]@tcp [ 3656.218289] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.202.19/24@tcp in log lustre-MDT0000 [ 3656.229255] Lustre: Skipped 2 previous similar messages [ 3665.518584] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.19/24@tcp [ 3665.524238] Lustre: Skipped 1 previous similar message [ 3672.040504] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3672.042057] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3672.049251] LustreError: Skipped 1 previous similar message [ 3672.062694] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3673.056626] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3673.066714] LustreError: Skipped 1 previous similar message [ 3673.068619] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3673.080749] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3675.121399] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3675.130743] Lustre: Skipped 1 previous similar message [ 3680.225149] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3680.240462] Lustre: Skipped 1 previous similar message [ 3685.350460] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3685.358664] Lustre: Skipped 1 previous similar message [ 3686.367159] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3686.417696] LustreError: 117904:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3686.589245] Lustre: server umount lustre-OST0000 complete [ 3694.053171] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3694.065133] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3694.085720] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3705.311518] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3705.514130] LustreError: 118106:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3705.524571] LustreError: 118106:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3705.656174] Lustre: server umount lustre-MDT0000 complete [ 3705.824969] LustreError: 117653:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3705.836976] LustreError: 117653:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3709.773464] LustreError: 117299:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768550247 with bad export cookie 11987400256336527325 [ 3709.778548] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3709.784290] LustreError: 117299:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3710.001033] LustreError: 118309:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 3710.008328] LustreError: 118309:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3710.306506] Lustre: server umount lustre-MDT0001 complete [ 3723.777367] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 3727.190443] Key type lgssc unregistered [ 3727.470813] LNet: 118840:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3727.481838] LNetError: 118840:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3727.511502] LNet: Removed LNI 192.168.202.119@tcp [ 3728.310552] Key type .llcrypt unregistered [ 3728.313073] Key type ._llcrypt unregistered [ 3760.550489] Key type ._llcrypt registered [ 3760.568963] Key type .llcrypt registered [ 3760.765103] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 3762.071114] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3762.293690] alg: No test for adler32 (adler32-zlib) [ 3763.603874] Lustre: Lustre: Build Version: 2.17.50_30_gbb2e37c [ 3763.969819] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 3765.727726] Key type lgssc registered [ 3767.123341] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3776.440707] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3776.465949] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3778.117259] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3778.150804] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.202.19/24@tcp [ 3778.155848] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3783.231446] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 3788.259695] LustreError: 120383:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3788.277968] LustreError: 120383:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3793.060318] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3793.434516] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3793.468567] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.19/24@tcp [ 3793.542245] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3799.092912] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 3805.747125] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3810.425519] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3818.163979] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3818.469226] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 3818.566601] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3822.662378] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 3825.774121] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 3832.431824] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3848.173779] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3848.187776] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3848.203864] Lustre: Skipped 1 previous similar message [ 3848.239496] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3851.422392] LustreError: 122681:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3851.782389] Lustre: server umount lustre-OST0000 complete [ 3858.401172] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3858.405201] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3858.406462] Lustre: Skipped 1 previous similar message [ 3858.419109] Lustre: Skipped 2 previous similar messages [ 3862.168254] LustreError: 122882:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3862.182422] LustreError: 122882:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3862.467156] Lustre: server umount lustre-MDT0000 complete [ 3863.520696] LustreError: 120383:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3863.543113] LustreError: 120383:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3867.279756] LustreError: 120362:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768550404 with bad export cookie 3698211780833289683 [ 3867.284288] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3867.299250] LustreError: 120362:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3867.527580] LustreError: 123085:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 3867.553095] LustreError: 123085:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3868.054264] Lustre: server umount lustre-MDT0001 complete [ 3880.717236] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 3885.056477] Key type lgssc unregistered [ 3885.540760] LNet: 123618:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3885.554395] LNetError: 123618:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3885.575583] LNet: Removed LNI 192.168.202.119@tcp [ 3886.670845] Key type .llcrypt unregistered [ 3886.673254] Key type ._llcrypt unregistered [ 3919.989372] Key type ._llcrypt registered [ 3919.991923] Key type .llcrypt registered [ 3920.141492] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [ 3936.053247] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 3937.410210] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3937.436690] alg: No test for adler32 (adler32-zlib) [ 3938.493924] Lustre: Lustre: Build Version: 2.17.50_30_gbb2e37c [ 3938.736413] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 3940.407186] Key type lgssc registered [ 3941.379853] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3976.073473] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3976.106621] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3977.423555] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3977.462069] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3977.583886] Lustre: lustre-MDT0000: new disk, initializing [ 3977.721287] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3977.742637] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3982.279484] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 3992.785871] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3992.935699] Lustre: 126997:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3993.030726] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3993.034429] Lustre: Skipped 1 previous similar message [ 3993.100558] Lustre: lustre-MDT0001: new disk, initializing [ 3993.253276] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3993.299157] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3993.329869] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3997.973918] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 4005.944813] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4009.867542] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4017.388864] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4017.787987] Lustre: lustre-OST0000: new disk, initializing [ 4017.790525] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4017.894708] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4023.375103] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 4023.384874] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4023.399700] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4023.499927] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4030.756430] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4035.761258] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4035.964672] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4040.450717] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4040.665354] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4044.268330] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4044.273320] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4044.275166] LustreError: Skipped 1 previous similar message [ 4044.276106] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4044.304175] Lustre: Skipped 1 previous similar message [ 4048.804190] LustreError: 129930:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4049.093119] Lustre: server umount lustre-OST0000 complete [ 4054.497671] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4054.523257] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4054.540090] Lustre: Skipped 2 previous similar messages [ 4059.617475] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4059.836621] LustreError: 130132:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4059.841195] LustreError: 130132:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4060.347078] Lustre: server umount lustre-MDT0000 complete [ 4064.803604] LustreError: 126988:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768550602 with bad export cookie 10982152017146929693 [ 4064.807042] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4064.811824] LustreError: 126988:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4064.946130] LustreError: 130333:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4064.953130] LustreError: 130333:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4065.231414] Lustre: server umount lustre-MDT0001 complete [ 4073.832186] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4074.217200] LustreError: 130809:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4074.229950] LustreError: 130809:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4074.289362] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4077.986269] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 4079.594523] LustreError: 130809:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4083.689167] LustreError: 130809:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4086.368178] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4086.714681] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4091.296717] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 4096.354328] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4100.423421] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4107.227209] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4107.532091] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4113.899393] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 4120.237414] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4125.139829] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4125.306187] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4130.085604] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4130.294681] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4138.897901] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4139.094625] Lustre: lustre-OST0001: new disk, initializing [ 4139.098665] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4139.206811] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4145.520267] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 4148.302761] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4148.309204] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4148.394289] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4154.911332] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4155.237575] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4160.353115] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4160.567676] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4164.683683] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff99f50578d800.ost_server_uuid 50 [ 4166.802805] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff99f50578d800.ost_server_uuid in FULL state after 0 sec [ 4178.562160] LustreError: 135049:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 4178.568996] LustreError: 135049:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4178.909364] Lustre: server umount lustre-OST0001 complete [ 4179.425159] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4179.430146] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4179.433351] LustreError: Skipped 1 previous similar message [ 4179.439583] Lustre: Skipped 1 previous similar message [ 4179.455163] LustreError: 132403:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4179.482746] LustreError: 132403:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4183.524111] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4183.532114] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4183.554555] Lustre: Skipped 1 previous similar message [ 4183.567790] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4183.570376] Lustre: Skipped 1 previous similar message [ 4184.549450] LustreError: 132914:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4184.568829] LustreError: 132914:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4189.664767] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4189.672231] Lustre: Skipped 2 previous similar messages [ 4194.794992] LustreError: 132402:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4194.811258] LustreError: 132402:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4197.343169] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4197.402474] LustreError: 135250:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4197.409103] LustreError: 135250:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4197.658612] Lustre: server umount lustre-OST0000 complete [ 4204.512722] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4204.520242] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4204.532570] Lustre: Skipped 1 previous similar message [ 4204.537248] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4204.543313] Lustre: Skipped 3 previous similar messages [ 4207.848795] LustreError: 135453:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 4207.858980] LustreError: 135453:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4207.989395] Lustre: server umount lustre-MDT0000 complete [ 4212.175802] LustreError: 130789:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768550749 with bad export cookie 10982152017146930575 [ 4212.182582] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4212.182636] LustreError: 130789:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4212.478903] Lustre: server umount lustre-MDT0001 complete [ 4231.152512] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 4234.741795] Key type lgssc unregistered [ 4235.010056] LNet: 136647:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4235.016585] LNetError: 136647:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4235.039543] LNet: Removed LNI 192.168.202.119@tcp [ 4235.870960] Key type .llcrypt unregistered [ 4235.873284] Key type ._llcrypt unregistered [ 4280.225489] Key type ._llcrypt registered [ 4280.228152] Key type .llcrypt registered [ 4280.442293] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [ 4295.767482] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 4297.128177] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4297.161349] alg: No test for adler32 (adler32-zlib) [ 4298.275804] Lustre: Lustre: Build Version: 2.17.50_30_gbb2e37c [ 4298.529123] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 4300.239535] Key type lgssc registered [ 4301.433382] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4340.000926] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4340.041564] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4341.485196] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4341.533450] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4341.631114] Lustre: lustre-MDT0000: new disk, initializing [ 4341.759987] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4341.785121] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4346.671983] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 4358.506739] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4358.688614] Lustre: 140767:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4358.772896] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4358.783610] Lustre: Skipped 1 previous similar message [ 4358.913349] Lustre: lustre-MDT0001: new disk, initializing [ 4359.059780] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4359.120487] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4359.144161] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4364.347411] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 4372.485551] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4376.340597] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4383.472631] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4383.715954] Lustre: lustre-OST0000: new disk, initializing [ 4383.722668] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4383.839659] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4390.016764] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 4393.029708] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4393.042118] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4393.262026] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4398.275112] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4403.905161] Lustre: Failing over lustre-OST0000 [ 4403.965104] LustreError: 143202:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4404.042523] Lustre: server umount lustre-OST0000 complete [ 4421.600091] Lustre: 138993:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1768550943/real 1768550943] req@0000000053516b99 x1854459960394752/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1768550959 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4421.617165] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4423.691748] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4424.197992] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4424.222632] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4424.233669] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4425.256216] Lustre: 138990:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1768550946/real 1768550946] req@0000000075f5cc78 x1854459960395008/t0(0) o13->lustre-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 1 dl 1768550962 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4425.312337] Lustre: 138990:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4426.286839] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4430.369428] Lustre: 138991:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1768550951/real 1768550951] req@00000000b60d8915 x1854459960396544/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1768550967 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4432.147028] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 4432.469332] Lustre: lustre-OST0000: Recovery over after 0:06, of 3 clients 3 recovered and 0 were evicted. [ 4432.473508] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4435.423364] Lustre: 138992:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1768550956/real 1768550956] req@0000000028b77bd8 x1854459960397824/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1768550972 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4435.462162] Lustre: 138992:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4436.647946] Lustre: Failing over lustre-MDT0000 [ 4436.822948] LustreError: 144187:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4436.833562] LustreError: 144187:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4436.958713] Lustre: server umount lustre-MDT0000 complete [ 4437.481782] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4437.490653] Lustre: Skipped 1 previous similar message [ 4441.944168] LustreError: 140773:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.19@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4441.970574] LustreError: 140773:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 4442.593719] LustreError: 142957:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4442.626170] LustreError: 142957:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4447.057861] LustreError: 142957:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.19@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4452.180838] LustreError: 140775:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.19@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4452.219665] LustreError: 140775:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4452.832142] Lustre: 138992:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1768550974/real 1768550974] req@000000007c549cd3 x1854459960408576/t0(0) o400->MGC192.168.202.119@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1768550990 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4452.875505] Lustre: 138992:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4452.889697] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4457.304222] LustreError: 140775:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.19@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4457.334525] LustreError: 140775:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4458.140394] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4463.073618] LustreError: 138989:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000009cf4e411 x1854459960413440/t0(0) o250->MGC192.168.202.119@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4463.469844] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4463.576038] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4466.594893] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4468.733537] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4468.739023] Lustre: Skipped 1 previous similar message [ 4468.773135] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 4468.901431] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4469.079930] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 4478.958235] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4478.968061] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4478.978160] Lustre: Skipped 1 previous similar message [ 4478.979539] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4483.969879] LustreError: 145075:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4483.978978] LustreError: 145075:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4484.141275] Lustre: server umount lustre-OST0000 complete [ 4489.188511] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4489.189630] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4489.197510] Lustre: Skipped 1 previous similar message [ 4489.199244] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4489.227615] Lustre: Skipped 2 previous similar messages [ 4494.305323] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4494.315519] Lustre: Skipped 1 previous similar message [ 4494.834911] LustreError: 145278:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 4494.849465] LustreError: 145278:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4495.391318] Lustre: server umount lustre-MDT0000 complete [ 4499.443336] LustreError: 140774:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4499.464987] LustreError: 140774:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 4500.730952] LustreError: 140759:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768551038 with bad export cookie 5440169410146407654 [ 4500.732939] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4500.737815] LustreError: 140759:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4500.907416] LustreError: 145480:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4500.915676] LustreError: 145480:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4501.277756] Lustre: server umount lustre-MDT0001 complete [ 4513.818758] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 4517.197803] Key type lgssc unregistered [ 4517.569178] LNet: 146014:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4517.597849] LNetError: 146014:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4518.641585] LNet: Removed LNI 192.168.202.119@tcp [ 4519.566713] Key type .llcrypt unregistered [ 4519.568989] Key type ._llcrypt unregistered [ 4557.767603] Key type ._llcrypt registered [ 4557.769802] Key type .llcrypt registered [ 4557.879399] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 4559.368981] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4559.625794] alg: No test for adler32 (adler32-zlib) [ 4560.857713] Lustre: Lustre: Build Version: 2.17.50_30_gbb2e37c [ 4561.182173] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 4562.878357] Key type lgssc registered [ 4564.311250] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4573.999587] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4574.054150] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4575.806639] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4581.345775] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 4585.954290] LustreError: 147515:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4585.970936] LustreError: 147515:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4590.409606] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4591.076351] LustreError: 147515:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4594.375992] LustreError: 147544:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4594.387751] LustreError: 147544:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4594.395377] LustreError: 147544:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 18, retries 0, failed: rc = -5 [ 4594.583278] LustreError: 148066:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4594.779552] Lustre: server umount lustre-MDT0000 complete [ 4611.926103] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [ 4631.508380] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 4671.360903] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4671.661419] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4671.720590] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4671.861814] Lustre: lustre-MDT0000: new disk, initializing [ 4671.991145] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4672.019781] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4676.380267] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 4689.150569] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4689.354669] Lustre: 150727:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4689.411145] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4689.414029] Lustre: Skipped 1 previous similar message [ 4689.495904] Lustre: lustre-MDT0001: new disk, initializing [ 4689.606838] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4689.656505] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4689.670674] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4694.479436] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 4703.421982] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4708.209170] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4716.866245] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4717.160638] Lustre: lustre-OST0000: new disk, initializing [ 4717.166607] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4717.268588] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4724.479777] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 4726.404322] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4726.417612] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4726.535226] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4732.268499] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4737.495471] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4740.185867] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 03:14:35 (1768551275) [ 4997.091316] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4997.102687] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4997.134497] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4998.112960] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4998.114783] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4998.145160] Lustre: Skipped 1 previous similar message [ 4998.157292] Lustre: Skipped 2 previous similar messages [ 4999.900930] LustreError: 154254:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4999.918969] LustreError: 154254:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5000.206969] Lustre: server umount lustre-MDT0000 complete [ 5003.235493] LustreError: 150738:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5003.253690] LustreError: 150738:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5004.997951] LustreError: 150719:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768551542 with bad export cookie 11178070391077985371 [ 5005.003916] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5005.023236] LustreError: 150719:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5008.355405] LustreError: 150733:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5008.360655] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5008.422159] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5011.276854] LustreError: 154457:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5011.287437] LustreError: 154457:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5011.478529] Lustre: server umount lustre-MDT0001 complete [ 5031.392053] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5031.456429] LustreError: 154661:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 5031.459952] LustreError: 154661:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5031.667285] Lustre: server umount lustre-OST0000 complete [ 5043.595088] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [ 5053.455068] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 5096.121225] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5096.500866] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5096.564444] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5096.736409] Lustre: lustre-MDT0000: new disk, initializing [ 5096.901284] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5096.919879] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5102.510573] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 5114.620410] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5114.789273] Lustre: 156991:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5114.860798] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5114.864245] Lustre: Skipped 1 previous similar message [ 5114.988652] Lustre: lustre-MDT0001: new disk, initializing [ 5115.185606] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5115.257600] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5115.284803] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5120.433419] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 5129.214966] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5133.976314] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5141.358755] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5141.635613] Lustre: lustre-OST0000: new disk, initializing [ 5141.642220] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5141.741736] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5149.000985] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 5149.815975] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5149.832798] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5150.021602] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5158.460983] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5163.566181] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5163.867986] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5169.643497] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5169.953361] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5175.776697] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5175.784877] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5175.788333] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5175.788344] Lustre: Skipped 1 previous similar message [ 5175.831528] Lustre: Skipped 1 previous similar message [ 5178.427277] LustreError: 159924:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5178.439258] LustreError: 159924:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5178.633290] Lustre: server umount lustre-OST0000 complete [ 5186.022163] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5186.045921] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5186.055895] Lustre: Skipped 2 previous similar messages [ 5189.574378] LustreError: 160125:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5189.582825] LustreError: 160125:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5189.813711] Lustre: server umount lustre-MDT0000 complete [ 5191.142076] LustreError: 157001:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5191.164800] LustreError: 157001:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5194.808209] LustreError: 156984:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768551732 with bad export cookie 11178070391078144208 [ 5194.821472] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5194.826644] LustreError: 156984:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5195.291943] Lustre: server umount lustre-MDT0001 complete [ 5206.105377] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 03:22:21 (1768551741) [ 5224.779567] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [ 5233.585842] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 5273.828483] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5274.163778] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5274.213394] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5274.354228] Lustre: lustre-MDT0000: new disk, initializing [ 5274.468296] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5274.505351] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5280.890121] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 5294.951862] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5295.218379] Lustre: 163308:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5295.621413] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5300.867491] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 5309.329210] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5313.498114] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5321.668447] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5322.043240] Lustre: lustre-OST0000: new disk, initializing [ 5322.045240] Lustre: Skipped 1 previous similar message [ 5322.053584] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5322.062570] Lustre: Skipped 2 previous similar messages [ 5327.971057] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5327.981647] Lustre: Skipped 1 previous similar message [ 5327.990167] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5328.119664] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5329.361176] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 5338.108397] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5348.839668] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5348.844039] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5348.846681] LustreError: Skipped 1 previous similar message [ 5348.849164] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5348.879424] Lustre: Skipped 2 previous similar messages [ 5351.371232] LustreError: 165852:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5351.374403] LustreError: 165852:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 5351.673537] Lustre: server umount lustre-OST0000 complete [ 5357.025214] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5361.901953] Lustre: server umount lustre-MDT0000 complete [ 5364.192167] LustreError: 163318:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5364.217456] LustreError: 163318:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5366.067170] LustreError: 165071:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768551903 with bad export cookie 11178070391078145090 [ 5366.070798] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5366.089674] LustreError: 165071:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5366.448487] Lustre: server umount lustre-MDT0001 complete [ 5375.298682] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 03:25:11 (1768551911) [ 5393.234086] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [ 5404.603800] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 5442.021760] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5442.251612] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5442.278036] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5442.382769] Lustre: lustre-MDT0000: new disk, initializing [ 5442.470118] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5442.476392] Lustre: Skipped 2 previous similar messages [ 5442.492413] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5446.922931] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 5458.516102] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5458.709846] Lustre: 169240:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5458.843577] LustreError: 169251:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5459.136411] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5463.739515] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 5471.526796] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5475.641587] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5482.787395] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5489.437508] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5489.990646] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 5497.942143] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5510.115799] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5510.116827] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5510.127377] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5510.127388] Lustre: Skipped 4 previous similar messages [ 5510.134736] Lustre: Skipped 3 previous similar messages [ 5512.890232] LustreError: 171778:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5512.903260] LustreError: 171778:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 5513.133623] Lustre: server umount lustre-OST0000 complete [ 5523.330956] Lustre: server umount lustre-MDT0000 complete [ 5525.481126] LustreError: 169251:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5525.495696] LustreError: 169251:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5527.075136] LustreError: 169232:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768552064 with bad export cookie 11178070391078146175 [ 5527.087585] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5527.092981] LustreError: 169232:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5527.555876] Lustre: server umount lustre-MDT0001 complete [ 5538.138479] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 5541.016489] Key type lgssc unregistered [ 5541.377366] LNet: 172714:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5541.390241] LNetError: 172714:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5541.420309] LNet: Removed LNI 192.168.202.119@tcp [ 5542.331350] Key type .llcrypt unregistered [ 5542.333610] Key type ._llcrypt unregistered [ 5565.362329] Key type ._llcrypt registered [ 5565.364393] Key type .llcrypt registered [ 5565.468155] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 5566.519294] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5566.683661] alg: No test for adler32 (adler32-zlib) [ 5567.755620] Lustre: Lustre: Build Version: 2.17.50_30_gbb2e37c [ 5568.028896] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 5569.655202] Key type lgssc registered [ 5570.843794] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5579.161741] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5579.185789] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5580.729051] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5585.515654] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 5591.016183] LustreError: 174091:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5591.053906] LustreError: 174091:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5594.947590] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5595.482929] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5600.935081] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 5607.404229] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5611.489441] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5619.314887] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5619.633356] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5625.827729] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5627.136489] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 5634.221205] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5651.433625] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5651.437306] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5651.458517] LustreError: Skipped 1 previous similar message [ 5651.466832] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5651.496494] Lustre: Skipped 1 previous similar message [ 5656.550583] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5656.563665] Lustre: Skipped 2 previous similar messages [ 5657.285105] LustreError: 176359:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5657.450362] Lustre: server umount lustre-OST0000 complete [ 5662.178576] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5662.183217] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5662.198824] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5666.785226] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5666.800649] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5668.022562] LustreError: 176560:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5668.030606] LustreError: 176560:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5668.150173] Lustre: server umount lustre-MDT0000 complete [ 5671.580284] LustreError: 174073:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768552209 with bad export cookie 15854483551941857528 [ 5671.585177] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5671.594985] LustreError: 174073:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5671.732405] LustreError: 176764:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5671.739125] LustreError: 176764:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5671.885634] Lustre: server umount lustre-MDT0001 complete [ 5684.928497] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 5688.118061] Key type lgssc unregistered [ 5688.412061] LNet: 177298:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5688.418574] LNetError: 177298:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5688.450530] LNet: Removed LNI 192.168.202.119@tcp [ 5689.294979] Key type .llcrypt unregistered [ 5689.296885] Key type ._llcrypt unregistered [ 5722.211167] Key type ._llcrypt registered [ 5722.219375] Key type .llcrypt registered [ 5722.417454] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 5723.562903] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5723.594355] alg: No test for adler32 (adler32-zlib) [ 5724.898234] Lustre: Lustre: Build Version: 2.17.50_30_gbb2e37c [ 5725.090830] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 5726.743190] Key type lgssc registered [ 5728.045722] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5735.809499] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5735.840688] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5737.395897] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5742.533571] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 5746.657663] LustreError: 178696:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5746.694844] LustreError: 178696:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5751.485753] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5751.865040] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5756.732229] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 5762.058673] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5765.880786] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5772.537618] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5772.865656] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5779.119057] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 5782.129454] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5785.166283] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5792.107937] Lustre: lustre-OST0000: Not available for connect from 192.168.202.19@tcp (stopping) [ 5792.738938] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5792.742128] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5792.744765] LustreError: Skipped 1 previous similar message [ 5792.745803] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5792.768140] Lustre: Skipped 1 previous similar message [ 5797.201436] Lustre: lustre-OST0000: Not available for connect from 192.168.202.19@tcp (stopping) [ 5797.219042] Lustre: Skipped 1 previous similar message [ 5797.520040] LustreError: 180937:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5797.722906] Lustre: server umount lustre-OST0000 complete [ 5803.614395] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff99f510d70800.ost_server_uuid 50 [ 5805.204310] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5823.458240] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5823.461083] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5823.489563] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5825.608359] LustreError: 181307:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5825.614457] LustreError: 181307:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5825.853760] Lustre: server umount lustre-MDT0000 complete [ 5828.579952] LustreError: 178696:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5828.594538] LustreError: 178696:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5830.787525] LustreError: 178677:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768552368 with bad export cookie 7751798768725831362 [ 5830.792706] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5830.808901] LustreError: 178677:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5831.004228] LustreError: 181509:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5831.019664] LustreError: 181509:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5831.634628] Lustre: server umount lustre-MDT0001 complete [ 5843.883464] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 03:32:59 (1768552379) [ 5852.568359] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5853.022771] LustreError: 182180:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5853.091788] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5857.479549] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 5858.273617] LustreError: 182180:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5863.396180] LustreError: 182180:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5866.591117] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5867.080917] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5871.846402] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 5877.023972] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5881.459591] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5890.116695] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5890.680529] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5897.922558] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5898.393944] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 5904.829978] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5912.350901] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5912.635392] Lustre: lustre-OST0001: new disk, initializing [ 5912.641266] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5912.758121] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5917.850972] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5917.871971] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5918.089681] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5920.862544] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 5931.417851] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5938.044576] Lustre: lustre-OST0000: Not available for connect from 192.168.202.19@tcp (stopping) [ 5938.050689] Lustre: Skipped 1 previous similar message [ 5938.153088] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5938.158528] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5938.172483] Lustre: Skipped 1 previous similar message [ 5941.439330] LustreError: 185379:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5941.443821] LustreError: 185379:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5941.495587] Lustre: server umount lustre-OST0000 complete [ 5943.121896] LustreError: 183775:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.19@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5946.946809] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5947.226257] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5953.285269] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5953.564397] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5959.138858] LustreError: 183776:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5959.164696] LustreError: 183776:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 10 previous similar messages [ 5969.385679] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5969.400518] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5969.410696] Lustre: Skipped 1 previous similar message [ 5969.415873] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5969.422406] Lustre: Skipped 3 previous similar messages [ 5973.961851] LustreError: 186107:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 5973.965468] LustreError: 186107:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5974.107773] Lustre: server umount lustre-OST0001 complete [ 5979.617226] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5979.617435] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5979.635196] Lustre: Skipped 1 previous similar message [ 5979.641154] LustreError: Skipped 1 previous similar message [ 5985.258921] LustreError: 186309:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 5985.267521] LustreError: 186309:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5985.459152] Lustre: server umount lustre-MDT0000 complete [ 5990.034579] LustreError: 182159:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768552527 with bad export cookie 7751798768725832440 [ 5990.039326] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5990.046336] LustreError: 182159:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5990.540694] Lustre: server umount lustre-MDT0001 complete [ 6018.793922] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 03:35:54 (1768552554) [ 6027.497297] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6027.858504] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6027.904475] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6028.093332] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6032.863968] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 6042.790316] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6042.894231] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6042.935675] Lustre: 187981:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6048.136924] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 6055.393610] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6060.436649] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6068.134208] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6068.285934] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6074.545131] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6075.779514] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 6081.844485] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6089.674989] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6089.767707] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6096.929110] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 6103.754191] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6109.566497] Lustre: lustre-OST0000: Not available for connect from 192.168.202.19@tcp (stopping) [ 6109.574874] Lustre: Skipped 4 previous similar messages [ 6110.179352] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6110.188740] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6110.206962] Lustre: Skipped 1 previous similar message [ 6113.103106] LustreError: 191168:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6113.106559] LustreError: 191168:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 6113.288359] Lustre: server umount lustre-OST0000 complete [ 6114.652750] LustreError: 189604:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.19@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6114.679805] LustreError: 189604:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 6132.194038] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6132.196156] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6132.209729] LustreError: Skipped 2 previous similar messages [ 6132.233091] Lustre: Skipped 2 previous similar messages [ 6136.774927] Lustre: server umount lustre-OST0001 complete [ 6142.441204] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6142.449430] Lustre: Skipped 7 previous similar messages [ 6147.289149] LustreError: 191595:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 6147.294256] LustreError: 191595:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6147.745573] Lustre: server umount lustre-MDT0000 complete [ 6152.437460] LustreError: 187971:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768552689 with bad export cookie 7751798768725833847 [ 6152.442686] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6152.448579] LustreError: 187971:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6152.814910] Lustre: server umount lustre-MDT0001 complete [ 6188.147386] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 03:38:43 (1768552723) [ 6207.406668] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [ 6219.933884] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 6261.885592] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6262.178427] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6262.241233] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6262.361485] Lustre: lustre-MDT0000: new disk, initializing [ 6262.415510] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6262.418129] Lustre: Skipped 3 previous similar messages [ 6262.435981] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6267.246296] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 6278.426607] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6278.632933] Lustre: 195590:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6278.714203] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6278.717221] Lustre: Skipped 1 previous similar message [ 6278.835187] Lustre: lustre-MDT0001: new disk, initializing [ 6279.072133] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6279.090937] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6283.637066] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 6292.884934] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6298.830100] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6307.771055] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6308.135389] Lustre: lustre-OST0000: new disk, initializing [ 6308.147407] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6316.175449] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6316.180322] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6316.318982] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6317.438869] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 6326.523238] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6330.401227] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6330.676678] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6336.438890] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6336.749725] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6342.117945] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6342.118314] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6342.138036] LustreError: Skipped 1 previous similar message [ 6342.140094] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6342.182816] Lustre: Skipped 3 previous similar messages [ 6344.646432] LustreError: 198532:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6344.649348] LustreError: 198532:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 6344.816319] Lustre: server umount lustre-OST0000 complete [ 6356.071990] Lustre: server umount lustre-MDT0000 complete [ 6357.473561] LustreError: 195604:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6357.511430] LustreError: 195604:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 15 previous similar messages [ 6360.228705] LustreError: 195584:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768552897 with bad export cookie 7751798768725835408 [ 6360.244464] LustreError: 195584:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6360.246504] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6360.714059] Lustre: server umount lustre-MDT0001 complete [ 6369.841145] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6376.441406] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 6386.487720] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6393.204574] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 6400.041911] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6405.143687] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6413.789158] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6414.356174] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6414.369139] Lustre: Skipped 4 previous similar messages [ 6420.839383] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 6426.191045] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6430.904855] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6431.228420] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6437.380824] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6437.639973] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6440.928867] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6440.936398] LustreError: Skipped 1 previous similar message [ 6440.940395] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6440.955287] Lustre: Skipped 2 previous similar messages [ 6454.239453] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6454.523866] Lustre: server umount lustre-OST0000 complete [ 6461.888411] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6466.722949] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6497.051699] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6504.157508] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 6506.684439] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6506.712442] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6506.716354] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6510.722818] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6514.753260] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 2 sec [ 6522.341058] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6522.350718] Lustre: Skipped 11 previous similar messages [ 6524.573267] LustreError: 203791:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6524.593379] LustreError: 203791:0:(obd_class.h:479:obd_check_dev()) Skipped 21 previous similar messages [ 6524.770533] Lustre: server umount lustre-OST0000 complete [ 6534.945560] Lustre: server umount lustre-MDT0000 complete [ 6537.697439] LustreError: 201067:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6537.734274] LustreError: 201067:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 6539.061564] LustreError: 201027:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768553076 with bad export cookie 7751798768725836290 [ 6539.064383] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6539.073956] LustreError: 201027:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6539.641498] Lustre: server umount lustre-MDT0001 complete [ 6550.438820] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 03:44:46 (1768553086) [ 6559.127887] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6565.646207] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 6575.299957] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6580.939132] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 6588.332291] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6592.437644] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6599.926068] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6607.326451] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 6614.129950] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6619.715755] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6620.035275] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6625.807681] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6626.024798] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6633.237323] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6633.478414] Lustre: lustre-OST0001: new disk, initializing [ 6633.483144] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6640.293459] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 6640.851621] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6640.860792] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6641.088349] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6650.491469] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6657.590369] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6657.850308] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6662.001374] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6662.268055] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6666.720455] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6666.722454] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6666.737711] LustreError: Skipped 5 previous similar messages [ 6666.760126] Lustre: Skipped 6 previous similar messages [ 6679.009713] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6679.151322] Lustre: server umount lustre-OST0001 complete [ 6686.171727] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6686.363868] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6690.054450] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6690.197656] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6719.758535] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6720.133597] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6720.141494] Lustre: Skipped 5 previous similar messages [ 6725.625888] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6725.655930] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6725.668928] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6725.688853] Lustre: Skipped 1 previous similar message [ 6726.610469] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 6732.323058] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6734.206333] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6751.200106] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6751.415549] Lustre: server umount lustre-OST0001 complete [ 6775.384240] Lustre: server umount lustre-OST0000 complete [ 6781.930550] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6781.933934] Lustre: Skipped 20 previous similar messages [ 6786.959820] LustreError: 211152:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 6786.965325] LustreError: 211152:0:(obd_class.h:479:obd_check_dev()) Skipped 29 previous similar messages [ 6787.170552] Lustre: server umount lustre-MDT0000 complete [ 6791.361428] LustreError: 204844:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768553328 with bad export cookie 7751798768725837256 [ 6791.377043] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6791.386879] LustreError: 204844:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6797.859674] Lustre: server umount lustre-MDT0001 complete [ 6824.643904] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 03:49:20 (1768553360) [ 6832.443874] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6833.019701] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6833.047806] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6838.020127] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 6848.027140] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6848.143081] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6848.185433] Lustre: 212828:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6852.957574] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 6859.116809] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6863.102476] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6869.901585] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6870.243888] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6877.958320] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 6885.458607] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6897.112586] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6897.290192] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6905.010860] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 6911.499997] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6913.663698] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6919.527459] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6919.866314] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6924.250344] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6924.477991] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6928.483979] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff99f511869000.ost_server_uuid 50 [ 6930.430939] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff99f511869000.ost_server_uuid in IDLE state after 0 sec [ 6931.788946] Lustre: Permanently deactivating lustre-OST0001 [ 6931.803559] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6936.279758] Lustre: Permanently reactivating lustre-OST0001 [ 6936.301874] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 6936.307937] Lustre: Skipped 2 previous similar messages [ 6942.185474] LustreError: lustre-OST0001-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 6942.200355] LustreError: Skipped 3 previous similar messages [ 6942.208213] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6942.216950] Lustre: Skipped 6 previous similar messages [ 6946.825395] Lustre: server umount lustre-OST0001 complete [ 6948.844402] LustreError: 214817:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6948.859275] LustreError: 214817:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 35 previous similar messages [ 6956.774070] Lustre: server umount lustre-OST0000 complete [ 6967.574294] Lustre: server umount lustre-MDT0000 complete [ 6971.432074] LustreError: 212819:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768553508 with bad export cookie 7751798768725838544 [ 6971.433142] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6971.441106] LustreError: 212819:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6971.691941] Lustre: server umount lustre-MDT0001 complete [ 6998.258732] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 03:52:14 (1768553534) [ 7006.504919] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7006.914506] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7006.945841] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7006.955491] Lustre: Skipped 2 previous similar messages [ 7012.109757] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 7021.400202] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7021.482522] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7021.540276] Lustre: 218939:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7027.173315] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 7034.116965] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7038.146494] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7046.067833] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7046.307954] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7046.341757] Lustre: Permanently deactivating lustre-OST0000 [ 7046.383959] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7052.356633] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7053.862580] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 7060.901245] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7068.876388] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7068.956737] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7074.611069] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7076.750659] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 7083.322351] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7085.522215] Lustre: Permanently reactivating lustre-OST0000 [ 7085.563025] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7085.571548] Lustre: Skipped 2 previous similar messages [ 7093.327121] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 41 seconds. I think it's dead, and I am evicting it. exp 00000000a1a6af13, cur 1768553630 deadline 1768553619 last 1768553589 [ 7093.330656] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7093.352624] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7093.372244] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7093.377857] Lustre: Skipped 1 previous similar message [ 7093.403776] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7105.603177] Lustre: server umount lustre-OST0001 complete [ 7115.053641] Lustre: server umount lustre-OST0000 complete [ 7125.366243] Lustre: server umount lustre-MDT0000 complete [ 7129.090539] LustreError: 218930:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768553666 with bad export cookie 7751798768725840182 [ 7129.100215] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7129.101979] LustreError: 218930:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7129.434812] Lustre: server umount lustre-MDT0001 complete [ 7140.437799] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 7142.983701] Key type lgssc unregistered [ 7143.323432] LNet: 223347:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7143.344249] LNetError: 223347:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7144.422503] LNet: Removed LNI 192.168.202.119@tcp [ 7145.070262] Key type .llcrypt unregistered [ 7145.073989] Key type ._llcrypt unregistered [ 7172.154854] Key type ._llcrypt registered [ 7172.157239] Key type .llcrypt registered [ 7172.344170] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 7173.827698] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7173.892218] alg: No test for adler32 (adler32-zlib) [ 7174.951994] Lustre: Lustre: Build Version: 2.17.50_30_gbb2e37c [ 7175.191435] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 7176.895194] Key type lgssc registered [ 7177.962229] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7188.203738] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7188.226325] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7189.786652] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7194.388432] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 7200.234617] LustreError: 224793:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7200.262252] LustreError: 224793:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7203.915353] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7204.002656] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7204.060829] Lustre: 224782:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7204.073456] Lustre: Permanently deactivating lustre-MDT0001 [ 7204.098405] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7204.510532] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7204.518632] LustreError: 224823:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 7204.531583] LustreError: 224823:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 7204.536435] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7204.546260] LustreError: 224823:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 14, retries 1, failed: rc = -108 [ 7209.044726] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 7214.596907] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7218.746899] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7228.041372] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7228.337199] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7229.356472] LustreError: 226409:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7234.532331] LustreError: 226463:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7235.579203] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7236.333401] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 7240.680705] LustreError: 226410:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7240.720485] LustreError: 226410:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7243.309053] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7245.797547] LustreError: 226408:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7245.824708] LustreError: 226408:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7250.248166] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7250.522651] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7255.533632] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7255.547931] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7256.729845] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 7263.443400] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7266.425543] Lustre: Permanently reactivating lustre-MDT0001 [ 7266.438933] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7266.445477] Lustre: Skipped 2 previous similar messages [ 7272.824917] Lustre: lustre-MDT0001: Client 9abf61b3-d437-4c98-8c6e-94f20ded4933 (at 192.168.202.19@tcp) reconnecting [ 7277.987493] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7280.008561] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7286.726117] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7287.038397] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7292.878712] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7293.214500] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7295.793974] Lustre: Permanently deactivating lustre-MDT0001 [ 7295.801142] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7295.803658] Lustre: Skipped 2 previous similar messages [ 7305.356549] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7314.915460] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7314.921500] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7314.930731] Lustre: Skipped 1 previous similar message [ 7314.935232] Lustre: Skipped 3 previous similar messages [ 7319.787995] LustreError: 228907:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 7320.034926] LustreError: 224792:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7320.041080] LustreError: 224792:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7320.090610] Lustre: server umount lustre-MDT0000 complete [ 7323.889215] LustreError: 224773:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768553861 with bad export cookie 11554865912915518959 [ 7323.891212] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7323.902987] LustreError: 224773:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7324.066152] LustreError: 229110:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 7324.070407] LustreError: 229110:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 7324.202592] Lustre: server umount lustre-MDT0001 complete [ 7330.082427] LustreError: 229312:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 7330.094554] LustreError: 229312:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 7330.157885] Lustre: server umount lustre-OST0000 complete [ 7346.253944] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 03:58:01 (1768553881) [ 7361.030829] LustreError: 230318:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 7361.037100] LustreError: 230318:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 7361.184286] Lustre: server umount lustre-OST0001 complete [ 7368.655645] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [ 7378.386028] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 7416.382460] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7416.766515] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7416.779033] Lustre: Skipped 2 previous similar messages [ 7416.814627] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7416.916299] Lustre: lustre-MDT0000: new disk, initializing [ 7417.027533] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7417.043233] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7421.694950] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 7432.557355] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7432.711659] Lustre: 232496:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7432.770646] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7432.775025] Lustre: Skipped 1 previous similar message [ 7432.832450] Lustre: lustre-MDT0001: new disk, initializing [ 7432.909673] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7432.971397] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7432.989458] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7437.782756] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 7446.301685] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7450.241855] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7456.966353] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7457.333932] Lustre: lustre-OST0000: new disk, initializing [ 7457.338412] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7457.452364] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7462.511064] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7462.526216] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7462.661346] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7464.341757] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 7471.776494] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7474.061295] LustreError: 233734:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7482.621685] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7482.802149] Lustre: lustre-OST0001: new disk, initializing [ 7482.805912] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7484.081723] LustreError: 233734:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7484.109375] LustreError: 233734:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7489.362415] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 7490.709473] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7490.732000] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7490.940194] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7494.188847] LustreError: 233734:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7499.207526] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7501.378217] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7506.406299] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7506.414537] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7506.423740] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7506.444849] Lustre: Skipped 3 previous similar messages [ 7509.445626] LustreError: 235979:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 7509.459503] LustreError: 235979:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7509.734324] Lustre: server umount lustre-OST0001 complete [ 7511.540410] LustreError: 234221:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7511.560430] LustreError: 234221:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7529.441115] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7529.447588] LustreError: Skipped 1 previous similar message [ 7529.450778] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7529.459329] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7529.462990] Lustre: Skipped 2 previous similar messages [ 7532.006484] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7532.027665] Lustre: Skipped 1 previous similar message [ 7533.246352] LustreError: 236213:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7533.248710] LustreError: 236213:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7533.379147] Lustre: server umount lustre-OST0000 complete [ 7540.714479] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7540.722659] LustreError: Skipped 1 previous similar message [ 7540.726351] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7540.735102] Lustre: Skipped 1 previous similar message [ 7540.740537] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7543.681635] Lustre: server umount lustre-MDT0000 complete [ 7547.360689] LustreError: 233589:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7547.389755] LustreError: 233589:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 7547.858693] LustreError: 234236:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768554085 with bad export cookie 11554865912915521556 [ 7547.863988] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7547.869471] LustreError: 234236:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7554.145365] LustreError: 236616:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 7554.150762] LustreError: 236616:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 7554.309716] Lustre: server umount lustre-MDT0001 complete [ 7564.989874] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 7568.063457] Key type lgssc unregistered [ 7568.397804] LNet: 237149:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7568.403883] LNetError: 237149:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7569.451300] LNet: Removed LNI 192.168.202.119@tcp [ 7570.422528] Key type .llcrypt unregistered [ 7570.424372] Key type ._llcrypt unregistered [ 7609.643198] Key type ._llcrypt registered [ 7609.645707] Key type .llcrypt registered [ 7609.770333] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 7611.110343] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7611.168748] alg: No test for adler32 (adler32-zlib) [ 7612.215392] Lustre: Lustre: Build Version: 2.17.50_30_gbb2e37c [ 7612.459370] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 7614.184167] Key type lgssc registered [ 7615.298562] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7624.891568] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7624.923699] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7626.395556] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7626.431141] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7626.651786] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7631.267045] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 7640.377618] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7640.500524] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7640.527665] Lustre: 239339:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7640.811165] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7645.530939] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 7651.166866] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7654.747710] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7662.616475] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7662.928568] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7663.137378] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7669.374583] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 7670.983521] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 7675.754600] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7732.196174] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7732.205331] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7732.220988] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7732.706956] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7732.713718] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7732.726995] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7733.959158] LustreError: 241696:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7734.102672] Lustre: server umount lustre-OST0000 complete [ 7739.154655] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7754.627586] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7755.231309] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7762.228709] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 7769.222634] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7772.601402] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7772.627358] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7772.643447] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7772.654873] Lustre: Skipped 1 previous similar message [ 7813.608515] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7813.631848] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7813.646152] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7813.655186] Lustre: Skipped 1 previous similar message [ 7816.266143] LustreError: 243360:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7816.276385] LustreError: 243360:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7816.499681] Lustre: server umount lustre-OST0000 complete [ 7823.848285] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7823.849350] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7823.870060] Lustre: Skipped 2 previous similar messages [ 7827.063567] LustreError: 243563:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 7827.076410] LustreError: 243563:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7827.224090] Lustre: server umount lustre-MDT0000 complete [ 7831.355644] LustreError: 240064:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768554368 with bad export cookie 7420690409787353595 [ 7831.362327] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7831.370784] LustreError: 240064:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7831.497210] LustreError: 243767:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7831.504240] LustreError: 243767:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7831.742251] Lustre: server umount lustre-MDT0001 complete [ 7842.615892] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 7845.231644] Key type lgssc unregistered [ 7845.562600] LNet: 244300:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7845.571229] LNetError: 244300:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7845.586168] LNet: Removed LNI 192.168.202.119@tcp [ 7846.530395] Key type .llcrypt unregistered [ 7846.533526] Key type ._llcrypt unregistered [ 7870.351991] Key type ._llcrypt registered [ 7870.367087] Key type .llcrypt registered [ 7870.534139] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 7871.409699] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7871.472474] alg: No test for adler32 (adler32-zlib) [ 7872.808768] Lustre: Lustre: Build Version: 2.17.50_30_gbb2e37c [ 7873.053393] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 7874.678493] Key type lgssc registered [ 7875.631833] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7883.738370] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7883.758358] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7885.286136] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7891.677822] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 7895.523632] LustreError: 245695:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7895.564600] LustreError: 245695:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7900.647096] LustreError: 245695:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7904.729570] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7905.146341] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7909.906781] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 7915.753305] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7920.371241] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7929.103665] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7929.727345] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7930.752099] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 7937.478436] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 7943.381941] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7971.827328] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7971.835605] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7971.850428] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7971.871279] Lustre: Skipped 1 previous similar message [ 7974.293149] LustreError: 248711:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7974.408956] Lustre: server umount lustre-OST0000 complete [ 7982.048189] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7982.061740] Lustre: Skipped 1 previous similar message [ 7982.075948] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7982.084467] Lustre: Skipped 1 previous similar message [ 7987.177858] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7987.182613] Lustre: Skipped 1 previous similar message [ 7992.287195] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7992.291616] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7992.308856] Lustre: Skipped 1 previous similar message [ 7992.431358] LustreError: 248912:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7992.439130] LustreError: 248912:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7992.693525] Lustre: server umount lustre-MDT0000 complete [ 7996.625832] LustreError: 245675:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768554534 with bad export cookie 9523975545090195153 [ 7996.647834] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7996.798731] LustreError: 249116:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 7996.809180] LustreError: 249116:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7997.039566] Lustre: server umount lustre-MDT0001 complete [ 8009.368685] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 8012.358122] Key type lgssc unregistered [ 8012.816679] LNet: 249647:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8012.837311] LNetError: 249647:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8013.874314] LNet: Removed LNI 192.168.202.119@tcp [ 8015.494501] Key type .llcrypt unregistered [ 8015.501097] Key type ._llcrypt unregistered [ 8036.202500] Key type ._llcrypt registered [ 8036.211385] Key type .llcrypt registered [ 8036.336964] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 8037.657040] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8037.686229] alg: No test for adler32 (adler32-zlib) [ 8038.989824] Lustre: Lustre: Build Version: 2.17.50_30_gbb2e37c [ 8039.186386] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 8040.876128] Key type lgssc registered [ 8042.301659] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8053.064427] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8053.105852] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8055.123729] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8061.743963] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 8065.516104] LustreError: 250857:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8065.544864] LustreError: 250857:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8070.637249] LustreError: 250857:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8073.231119] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8073.886128] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8078.467787] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 8085.107569] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8089.430533] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8097.325690] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8097.921411] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8099.960497] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8105.770680] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 8114.152172] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8130.030995] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8130.036294] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8130.062822] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8131.048560] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8131.064481] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8131.067333] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8134.156106] LustreError: 253245:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8134.246811] Lustre: server umount lustre-OST0000 complete [ 8140.260424] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8140.264750] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8140.304300] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8140.310349] Lustre: Skipped 1 previous similar message [ 8146.400474] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8146.409893] Lustre: Skipped 3 previous similar messages [ 8151.533903] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8151.539615] Lustre: Skipped 1 previous similar message [ 8153.055134] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8153.163418] LustreError: 253447:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 8153.166266] LustreError: 253447:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8153.536703] Lustre: server umount lustre-MDT0000 complete [ 8155.621615] LustreError: 251570:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8155.636543] LustreError: 251570:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8157.159079] LustreError: 250838:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768554694 with bad export cookie 12968349875053230507 [ 8157.165670] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8157.370496] LustreError: 253650:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 8157.378330] LustreError: 253650:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8157.600202] Lustre: server umount lustre-MDT0001 complete [ 8168.569313] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 8171.405984] Key type lgssc unregistered [ 8171.773747] LNet: 254184:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8171.780572] LNetError: 254184:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8171.798211] LNet: Removed LNI 192.168.202.119@tcp [ 8172.870328] Key type .llcrypt unregistered [ 8172.877684] Key type ._llcrypt unregistered [ 8198.998940] Key type ._llcrypt registered [ 8199.007382] Key type .llcrypt registered [ 8199.186565] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 8201.031837] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8201.127233] alg: No test for adler32 (adler32-zlib) [ 8202.441899] Lustre: Lustre: Build Version: 2.17.50_30_gbb2e37c [ 8202.745834] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 8204.423919] Key type lgssc registered [ 8206.051929] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8216.330595] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8216.360479] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8217.711136] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8223.929702] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 8228.325413] LustreError: 255583:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8228.341510] LustreError: 255583:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8233.444522] LustreError: 255583:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8235.459697] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8235.815129] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8241.053485] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 8246.143249] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8250.339267] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8257.750400] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8258.227666] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8263.362898] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8264.859307] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 8270.938698] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8297.442702] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8297.452193] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8297.466606] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8299.488823] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8299.503075] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8299.503354] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8299.517784] Lustre: Skipped 1 previous similar message [ 8303.555829] LustreError: 258599:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8303.747986] Lustre: server umount lustre-OST0000 complete [ 8309.737655] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8309.755860] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8309.772874] Lustre: Skipped 1 previous similar message [ 8309.787248] Lustre: Skipped 1 previous similar message [ 8314.454895] LustreError: 258801:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 8314.483409] LustreError: 258801:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8314.782860] Lustre: server umount lustre-MDT0000 complete [ 8314.849375] LustreError: 255579:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8314.893402] LustreError: 255579:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8319.613957] LustreError: 255562:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768554857 with bad export cookie 1617789390296801963 [ 8319.617164] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8319.624543] LustreError: 255562:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8319.854635] LustreError: 259003:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 8319.864174] LustreError: 259003:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8320.131395] Lustre: server umount lustre-MDT0001 complete [ 8333.716985] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 8336.843895] Key type lgssc unregistered [ 8337.198196] LNet: 259534:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8337.209968] LNetError: 259534:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8337.244060] LNet: Removed LNI 192.168.202.119@tcp [ 8338.227931] Key type .llcrypt unregistered [ 8338.230294] Key type ._llcrypt unregistered [ 8359.720522] Key type ._llcrypt registered [ 8359.725462] Key type .llcrypt registered [ 8359.906431] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 8361.667890] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8361.709165] alg: No test for adler32 (adler32-zlib) [ 8362.861653] Lustre: Lustre: Build Version: 2.17.50_30_gbb2e37c [ 8363.141683] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 8364.839861] Key type lgssc registered [ 8366.716618] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8376.196159] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8376.210194] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8378.011849] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8383.780239] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 8387.579054] LustreError: 260767:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8387.591594] LustreError: 260767:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8392.677859] LustreError: 260767:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8393.950297] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8394.489442] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8399.813891] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 8406.116199] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8410.485670] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8418.242972] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8418.571734] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8422.709123] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8425.863730] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 8431.726357] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8448.489534] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8448.497083] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8448.509340] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8448.534521] Lustre: Skipped 1 previous similar message [ 8448.554380] Lustre: Skipped 1 previous similar message [ 8451.656199] LustreError: 263165:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8451.837869] Lustre: server umount lustre-OST0000 complete [ 8458.725467] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8458.728166] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8458.750793] Lustre: Skipped 1 previous similar message [ 8458.764466] Lustre: Skipped 1 previous similar message [ 8462.512887] LustreError: 263366:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 8462.518250] LustreError: 263366:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8462.629110] Lustre: server umount lustre-MDT0000 complete [ 8463.844425] LustreError: 260745:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8463.868345] LustreError: 260745:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8466.353995] LustreError: 263367:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768555003 with bad export cookie 10467756574945715167 [ 8466.356205] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8466.362796] LustreError: 263367:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8466.506416] LustreError: 263569:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 8466.517381] LustreError: 263569:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8466.809879] Lustre: server umount lustre-MDT0001 complete [ 8477.419041] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 8479.922349] Key type lgssc unregistered [ 8480.203189] LNet: 264101:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8480.210361] LNetError: 264101:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8480.226828] LNet: Removed LNI 192.168.202.119@tcp [ 8481.071411] Key type .llcrypt unregistered [ 8481.073548] Key type ._llcrypt unregistered [ 8493.124277] Key type ._llcrypt registered [ 8493.127101] Key type .llcrypt registered [ 8493.256635] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8511.761510] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [ 8528.087514] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 8529.202409] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8529.223526] alg: No test for adler32 (adler32-zlib) [ 8530.443802] Lustre: Lustre: Build Version: 2.17.50_30_gbb2e37c [ 8530.772678] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 8532.503303] Key type lgssc registered [ 8534.032880] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8573.757806] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8573.829139] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8575.411779] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8575.461600] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8575.667674] Lustre: lustre-MDT0000: new disk, initializing [ 8575.913872] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8575.985156] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8582.019344] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 8597.884721] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8598.097618] Lustre: 267598:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8598.196959] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8598.208355] Lustre: Skipped 1 previous similar message [ 8598.279711] Lustre: lustre-MDT0001: new disk, initializing [ 8598.495658] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8598.561366] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8598.603825] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8603.864801] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 8612.571826] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8616.947538] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8625.152848] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8625.350810] Lustre: lustre-OST0000: new disk, initializing [ 8625.354393] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8625.434352] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8630.914423] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8630.927375] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8631.099429] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8633.328472] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 8641.833582] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8647.389081] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8647.784033] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8652.967730] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8653.189387] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8656.864352] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8656.870529] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8656.879036] LustreError: Skipped 1 previous similar message [ 8656.898858] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8656.908525] Lustre: Skipped 1 previous similar message [ 8661.629230] LustreError: 270532:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 8661.761275] Lustre: server umount lustre-OST0000 complete [ 8666.145098] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8666.154885] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8666.161419] Lustre: Skipped 2 previous similar messages [ 8666.191451] Lustre: Skipped 1 previous similar message [ 8671.201706] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8671.213129] Lustre: Skipped 1 previous similar message [ 8672.438236] LustreError: 270733:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 8672.450740] LustreError: 270733:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8672.709739] Lustre: server umount lustre-MDT0000 complete [ 8677.013107] LustreError: 268377:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768555214 with bad export cookie 14975852808522940985 [ 8677.024677] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8677.029825] LustreError: 268377:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8677.242669] LustreError: 270937:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8677.248793] LustreError: 270937:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8677.445739] Lustre: server umount lustre-MDT0001 complete [ 8688.438362] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 04:20:23 (1768555223) [ 8695.626962] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8696.165520] LustreError: 271609:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8696.180297] LustreError: 271609:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 8696.243993] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8701.078358] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 8701.411688] LustreError: 271609:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8706.547189] LustreError: 271609:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8710.618330] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8710.887591] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8715.611946] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 8721.157685] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8724.608883] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8731.594623] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8731.954726] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8740.151827] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 8746.731247] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8761.825101] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8761.832666] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8761.838202] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8761.861698] Lustre: Skipped 1 previous similar message [ 8763.645628] LustreError: 273853:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8763.658475] LustreError: 273853:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 8764.028855] Lustre: server umount lustre-OST0000 complete [ 8772.068863] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8772.099952] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8772.111481] Lustre: Skipped 2 previous similar messages [ 8782.309554] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8782.320297] Lustre: Skipped 3 previous similar messages [ 8783.327587] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8783.488584] LustreError: 274055:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 8783.496640] LustreError: 274055:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8783.636719] Lustre: server umount lustre-MDT0000 complete [ 8787.425186] LustreError: 273733:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8787.454481] LustreError: 273733:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8789.491223] LustreError: 271590:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768555326 with bad export cookie 14975852808522941867 [ 8789.496980] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8789.503665] LustreError: 271590:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8790.015969] Lustre: server umount lustre-MDT0001 complete [ 8802.615194] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 8806.339534] Key type lgssc unregistered [ 8806.784534] LNet: 274792:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8806.796365] LNetError: 274792:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8806.814473] LNet: Removed LNI 192.168.202.119@tcp [ 8808.022396] Key type .llcrypt unregistered [ 8808.024067] Key type ._llcrypt unregistered [ 8839.204746] Key type ._llcrypt registered [ 8839.206319] Key type .llcrypt registered [ 8839.318238] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [ 8854.573974] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 8855.946117] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8856.005715] alg: No test for adler32 (adler32-zlib) [ 8857.122408] Lustre: Lustre: Build Version: 2.17.50_30_gbb2e37c [ 8857.371375] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 8859.127222] Key type lgssc registered [ 8860.618917] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8916.313334] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8916.349087] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8917.689848] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8917.713498] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8917.830453] Lustre: lustre-MDT0000: new disk, initializing [ 8917.959350] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8918.001776] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8922.866422] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 8934.154611] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8934.329323] Lustre: 278582:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8934.390339] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8934.394039] Lustre: Skipped 1 previous similar message [ 8934.465367] Lustre: lustre-MDT0001: new disk, initializing [ 8934.576632] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8934.635791] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8934.660264] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8939.321778] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 8946.874789] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8950.342968] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8958.224577] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8958.597986] Lustre: lustre-OST2710: new disk, initializing [ 8958.607396] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 8958.733877] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 8964.787138] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 8964.829071] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 8965.043452] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 8966.401541] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 8976.445985] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 8985.756889] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8985.993380] Lustre: lustre-OST03e8: new disk, initializing [ 8986.000247] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 8986.122050] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 8992.377701] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 8992.402416] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 8992.489262] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 8993.901343] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 9000.993945] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 9009.316615] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 9009.665098] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9014.380635] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 9014.700390] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9020.533760] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 9020.754329] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9025.799774] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 9026.039990] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9032.180379] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9032.194497] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9032.216425] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9033.085504] LustreError: 282921:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 9033.592434] Lustre: server umount lustre-MDT0000 complete [ 9033.714381] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9033.745058] Lustre: Skipped 2 previous similar messages [ 9037.990460] LustreError: 279361:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768555575 with bad export cookie 16495953635121936632 [ 9038.002402] LustreError: 279361:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9038.003727] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9038.221774] LustreError: 283123:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 9038.231800] LustreError: 283123:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 9038.498898] Lustre: server umount lustre-MDT0001 complete [ 9053.041100] LustreError: 283324:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 9053.047411] LustreError: 283324:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 9053.280837] Lustre: server umount lustre-OST2710 complete [ 9055.199629] Lustre: 276380:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1768555576/real 1768555576] req@000000001a87f436 x1854464740889472/t0(0) o400->lustre-MDT0001-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1768555592 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9055.253892] Lustre: lustre-MDT0001-lwp-OST03e8: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9057.864207] LustreError: 283526:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 9057.874537] LustreError: 283526:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 9058.098219] Lustre: server umount lustre-OST03e8 complete [ 9080.031044] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [ 9091.058839] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 9132.981027] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9133.763404] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9133.842757] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9134.002715] Lustre: lustre-MDT0000: new disk, initializing [ 9134.300082] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9134.388410] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9141.223439] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 9154.027394] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9154.182301] Lustre: 286313:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9154.239893] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9154.243292] Lustre: Skipped 1 previous similar message [ 9154.301694] Lustre: lustre-MDT0001: new disk, initializing [ 9154.395995] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9154.454231] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9154.466287] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9160.657786] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 9170.797546] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9174.653465] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9183.585432] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9183.955475] Lustre: lustre-OST0000: new disk, initializing [ 9183.964151] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9184.104615] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9191.997324] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 9193.060743] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9193.085088] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9193.244152] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9201.555395] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9206.962968] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9207.301776] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9211.967976] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9212.244107] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9219.055261] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9219.058975] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9219.069693] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9219.078112] LustreError: Skipped 1 previous similar message [ 9219.085258] Lustre: Skipped 1 previous similar message [ 9220.606145] LustreError: 289248:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 9220.614902] LustreError: 289248:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 9220.867926] Lustre: server umount lustre-OST0000 complete [ 9226.213601] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9226.229836] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9226.250784] Lustre: Skipped 1 previous similar message [ 9226.260894] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9229.285045] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9231.076374] LustreError: 289449:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 9231.079520] LustreError: 289449:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 9231.309937] Lustre: server umount lustre-MDT0000 complete [ 9234.404590] LustreError: 286318:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9234.418706] LustreError: 286318:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 9235.797038] LustreError: 286304:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768555773 with bad export cookie 16495953635121938312 [ 9235.807021] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9236.241654] Lustre: server umount lustre-MDT0001 complete [ 9246.874246] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 04:29:42 (1768555782) [ 9248.564598] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 9250.473804] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 04:29:46 (1768555786) [ 9254.925044] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 9285.136085] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9285.425434] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9285.446357] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9285.583205] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9289.829318] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 9298.651377] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9298.727148] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9298.761313] Lustre: 291749:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9304.295226] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 9309.930489] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9313.443879] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9332.981414] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9333.218975] Lustre: Denying initial registration attempt from nid 192.168.202.119@tcp, specified as failover [ 9333.229595] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9333.239744] LustreError: 293362:0:(tgt_mount.c:2483:server_fill_super()) Unable to start targets: -99 [ 9333.254196] LustreError: 293362:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [ 9333.262258] LustreError: 293362:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 9333.341825] Lustre: server umount lustre-OST0000 complete [ 9333.345780] LustreError: 293362:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9341.926100] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9341.927968] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9341.944720] Lustre: Skipped 2 previous similar messages [ 9341.947499] Lustre: Skipped 1 previous similar message [ 9347.617068] LustreError: 293729:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 9347.619727] LustreError: 293729:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 9347.726301] Lustre: server umount lustre-MDT0000 complete [ 9351.924797] LustreError: 293730:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768555889 with bad export cookie 16495953635121939194 [ 9351.928717] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9351.938676] LustreError: 293730:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 9352.282084] Lustre: server umount lustre-MDT0001 complete [ 9362.158592] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 9364.723356] Key type lgssc unregistered [ 9365.052649] LNet: 294465:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9365.058532] LNetError: 294465:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9365.079220] LNet: Removed LNI 192.168.202.119@tcp [ 9365.842886] Key type .llcrypt unregistered [ 9365.844835] Key type ._llcrypt unregistered [ 9378.384477] Key type ._llcrypt registered [ 9378.389924] Key type .llcrypt registered [ 9378.547499] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 9380.384676] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9380.431471] alg: No test for adler32 (adler32-zlib) [ 9381.480408] Lustre: Lustre: Build Version: 2.17.50_30_gbb2e37c [ 9381.669391] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 9383.336194] Key type lgssc registered [ 9384.550551] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9442.901227] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 9453.378545] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9453.415075] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9454.835275] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9454.883137] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9455.086506] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9459.417500] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 9469.296922] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9469.419481] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9469.459185] Lustre: 297201:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9469.734654] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9474.952088] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 9481.826239] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9485.595812] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9503.827804] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9504.014469] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9504.254127] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9511.407930] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 9518.020691] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9521.634939] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9521.637669] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9521.648079] LustreError: Skipped 1 previous similar message [ 9521.650208] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9521.659235] Lustre: Skipped 1 previous similar message [ 9521.673332] Lustre: Skipped 1 previous similar message [ 9526.546209] LustreError: 299481:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 9526.707918] Lustre: server umount lustre-OST0000 complete [ 9531.361873] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9531.368527] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9531.377912] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9536.511543] LustreError: 299684:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 9536.520330] LustreError: 299684:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 9536.698844] Lustre: server umount lustre-MDT0000 complete [ 9539.894822] LustreError: 297192:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768556077 with bad export cookie 4397868298400392921 [ 9539.897517] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9539.902046] LustreError: 297192:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9540.015035] LustreError: 299885:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 9540.021636] LustreError: 299885:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 9540.254679] Lustre: server umount lustre-MDT0001 complete [ 9550.644233] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 9553.895384] Key type lgssc unregistered [ 9554.228569] LNet: 300417:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9554.241741] LNetError: 300417:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9554.268234] LNet: Removed LNI 192.168.202.119@tcp [ 9555.318169] Key type .llcrypt unregistered [ 9555.320447] Key type ._llcrypt unregistered [ 9578.935550] Key type ._llcrypt registered [ 9578.938265] Key type .llcrypt registered [ 9579.056757] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 9580.392538] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9580.419393] alg: No test for adler32 (adler32-zlib) [ 9581.461716] Lustre: Lustre: Build Version: 2.17.50_30_gbb2e37c [ 9581.659736] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 9583.287672] Key type lgssc registered [ 9584.386210] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9594.181424] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9594.228714] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9595.907964] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9600.220468] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 9606.119973] LustreError: 301796:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9606.147663] LustreError: 301796:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9609.930182] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9610.405393] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9615.794624] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 9623.101587] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9628.020121] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9635.546865] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9636.103560] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9645.961564] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 9653.816304] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9664.995683] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9665.000557] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9665.009311] Lustre: Skipped 2 previous similar messages [ 9665.015517] Lustre: Skipped 1 previous similar message [ 9666.424336] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.19@tcp (stopping) [ 9666.434540] Lustre: Skipped 1 previous similar message [ 9669.216877] LustreError: 304042:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 9669.520352] Lustre: server umount lustre-MDT0000 complete [ 9670.126457] LustreError: 303903:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9670.150172] LustreError: 303903:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9671.517128] LustreError: 301791:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.19@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9673.496812] LustreError: 301777:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768556210 with bad export cookie 3591204655703845125 [ 9673.521322] LustreError: 301777:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9673.521791] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9675.233863] LustreError: 301793:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9675.238264] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9675.288809] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9678.252558] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9678.259135] Lustre: Skipped 1 previous similar message [ 9679.744551] LustreError: 304245:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 9679.760653] LustreError: 304245:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 9680.084624] Lustre: server umount lustre-MDT0001 complete [ 9687.683684] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9698.326334] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9718.773148] LustreError: 303403:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9718.805459] LustreError: 303403:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9718.951898] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9719.108702] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 9723.858334] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 9728.495679] LustreError: 305055:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9728.525615] LustreError: 305055:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9732.597273] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9734.576428] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9735.234260] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9740.292269] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9740.719970] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 9746.956573] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9748.407645] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9752.941889] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9754.638336] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9759.724040] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9759.733963] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9759.746227] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9760.736900] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9765.435188] LustreError: 306549:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 9765.438170] LustreError: 306549:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 9765.529981] Lustre: server umount lustre-OST0000 complete [ 9770.978100] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9770.978293] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9770.998083] Lustre: Skipped 1 previous similar message [ 9771.000962] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9771.024380] Lustre: Skipped 3 previous similar messages [ 9775.949953] LustreError: 306750:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 9775.959366] LustreError: 306750:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 9776.094751] Lustre: server umount lustre-MDT0000 complete [ 9776.099489] LustreError: 305060:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9776.131837] LustreError: 305060:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9779.765236] LustreError: 301776:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768556317 with bad export cookie 3591204655703852650 [ 9779.781654] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9779.785834] LustreError: 301776:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9779.948174] LustreError: 306952:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 9779.950718] LustreError: 306952:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 9780.149470] Lustre: server umount lustre-MDT0001 complete [ 9794.114916] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 9797.146243] Key type lgssc unregistered [ 9797.433950] LNet: 307484:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9797.441706] LNetError: 307484:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9798.503184] LNet: Removed LNI 192.168.202.119@tcp [ 9799.436229] Key type .llcrypt unregistered [ 9799.439758] Key type ._llcrypt unregistered [ 9825.497523] Key type ._llcrypt registered [ 9825.501600] Key type .llcrypt registered [ 9825.722413] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 9827.897137] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9827.931373] alg: No test for adler32 (adler32-zlib) [ 9829.132435] Lustre: Lustre: Build Version: 2.17.50_30_gbb2e37c [ 9829.472824] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 9831.183598] Key type lgssc registered [ 9832.617459] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9841.653209] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9841.696102] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9843.184336] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9849.052950] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 9853.417876] LustreError: 308889:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9853.446086] LustreError: 308889:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9858.531207] LustreError: 308889:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9859.215483] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9859.555339] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9863.999113] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 9870.215088] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9875.243141] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9879.010472] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9879.029176] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9884.135691] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9884.138960] Lustre: Skipped 2 previous similar messages [ 9889.252343] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9889.267930] Lustre: Skipped 1 previous similar message [ 9893.345196] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9893.464187] LustreError: 310279:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 9893.596533] Lustre: server umount lustre-MDT0000 complete [ 9894.373335] LustreError: 308886:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9894.395894] LustreError: 308886:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9898.241712] LustreError: 308870:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768556435 with bad export cookie 1406726844065227917 [ 9898.254596] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9898.405244] LustreError: 310482:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 9898.413266] LustreError: 310482:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 9898.571892] Lustre: server umount lustre-MDT0001 complete [ 9916.706262] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9917.009988] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9917.050343] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9917.268625] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9922.498173] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 9931.995097] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9932.110141] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9932.155213] Lustre: 311252:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9932.347774] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9936.867536] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 9942.643512] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9947.328722] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9956.709050] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9956.984513] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 9957.005252] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 9957.018525] LustreError: 312908:0:(tgt_mount.c:2483:server_fill_super()) Unable to start targets: -2 [ 9957.024405] LustreError: 312908:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [ 9957.032410] LustreError: 312908:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 9957.093439] Lustre: server umount lustre-OST0000 complete [ 9957.096472] LustreError: 312908:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 9967.079138] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9967.256908] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9967.411488] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9974.736718] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 9974.941643] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 9979.964645] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9988.384576] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9988.530211] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 9988.606861] Lustre: lustre-OST0001: new disk, initializing [ 9988.611072] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9988.749573] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9996.618274] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 9997.995681] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9998.002749] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9998.149776] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10003.180929] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [10008.544605] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10008.551784] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10008.557777] LustreError: Skipped 1 previous similar message [10008.572886] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10008.580327] Lustre: Skipped 2 previous similar messages [10008.600478] Lustre: Skipped 1 previous similar message [10012.299705] LustreError: 315028:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [10012.309650] LustreError: 315028:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [10012.396680] Lustre: server umount lustre-OST0001 complete [10013.672861] LustreError: 313742:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10013.701534] LustreError: 313742:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10016.225396] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10016.238173] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10016.250528] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10018.793656] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10018.807370] LustreError: 313742:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10018.820318] LustreError: 313742:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10022.212497] LustreError: 315231:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [10022.215101] LustreError: 315231:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [10022.298462] Lustre: server umount lustre-OST0000 complete [10029.025109] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10029.027489] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10029.050817] Lustre: Skipped 1 previous similar message [10029.057887] Lustre: Skipped 2 previous similar messages [10032.468237] LustreError: 315432:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [10032.476543] LustreError: 315432:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [10032.697517] Lustre: server umount lustre-MDT0000 complete [10034.158912] LustreError: 311265:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10034.185510] LustreError: 311265:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10037.018596] LustreError: 311244:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768556574 with bad export cookie 1406726844065228470 [10037.021265] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10037.029323] LustreError: 311244:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [10037.427213] Lustre: server umount lustre-MDT0001 complete [10048.993912] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [10051.643769] Key type lgssc unregistered [10051.902311] LNet: 316166:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10051.906846] LNetError: 316166:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [10052.967800] LNet: Removed LNI 192.168.202.119@tcp [10053.873064] Key type .llcrypt unregistered [10053.874281] Key type ._llcrypt unregistered [10133.564367] Key type ._llcrypt registered [10133.567972] Key type .llcrypt registered [10133.671728] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [10148.436557] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [10149.581145] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10149.625866] alg: No test for adler32 (adler32-zlib) [10150.645204] Lustre: Lustre: Build Version: 2.17.50_30_gbb2e37c [10150.865626] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [10152.560654] Key type lgssc registered [10153.650658] Lustre: Echo OBD driver; http://www.lustre.org/ [10188.203535] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10188.224613] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10189.464597] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10189.491447] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10189.548457] Lustre: lustre-MDT0000: new disk, initializing [10189.642438] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10189.661382] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10193.980152] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [10206.329132] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10206.507712] Lustre: 321067:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10206.560196] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10206.563508] Lustre: Skipped 1 previous similar message [10206.626205] Lustre: lustre-MDT0001: new disk, initializing [10206.741231] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10206.807036] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10206.823768] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10212.140710] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [10220.829289] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10225.058098] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10234.069238] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10234.623225] Lustre: lustre-OST0000: new disk, initializing [10234.628233] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10234.788674] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10240.676760] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10240.706826] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10241.009776] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10242.684232] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [10251.275530] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10256.573652] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10256.818601] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10261.829852] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10262.068744] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10266.592618] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10266.595918] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10266.606190] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10266.632205] Lustre: Skipped 1 previous similar message [10270.547817] LustreError: 324001:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [10270.819914] Lustre: server umount lustre-OST0000 complete [10276.833096] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10276.852159] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10276.861296] Lustre: Skipped 2 previous similar messages [10281.180316] LustreError: 324203:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [10281.188113] LustreError: 324203:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [10281.300277] Lustre: server umount lustre-MDT0000 complete [10285.515507] LustreError: 321059:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768556822 with bad export cookie 14321734067691878953 [10285.517975] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10285.524633] LustreError: 321059:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [10285.772432] LustreError: 324404:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [10285.777626] LustreError: 324404:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [10286.201062] Lustre: server umount lustre-MDT0001 complete [10296.294956] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 04:47:11 (1768556831) [10305.551663] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 04:47:21 (1768556841) [10315.120574] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10315.686650] LustreError: 325418:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10315.717637] LustreError: 325418:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10315.758217] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10320.570278] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [10320.873030] LustreError: 325418:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10325.990375] LustreError: 325418:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10330.387202] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10330.818069] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10335.703563] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [10342.005914] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10346.440574] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10354.001487] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10354.470283] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10361.050565] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [10366.784446] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10374.731796] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [10376.312779] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [10377.796412] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [10379.735973] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [10381.786447] Lustre: Failing over lustre-MDT0000 [10381.926042] LustreError: 327855:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [10381.928834] LustreError: 327855:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [10382.092402] Lustre: server umount lustre-MDT0000 complete [10382.309617] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10382.320056] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10382.335017] Lustre: Skipped 1 previous similar message [10382.343624] LustreError: 325417:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10386.401039] LustreError: 325417:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10386.406201] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10386.432190] LustreError: 325417:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10394.962634] LustreError: 325414:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.19@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10394.970353] LustreError: 325414:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [10402.783353] Lustre: 319290:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1768556923/real 1768556923] req@000000007d9aab63 x1854466097745408/t0(0) o400->MGC192.168.202.119@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1768556939 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10402.842347] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10403.382680] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10411.355868] LustreError: 325412:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.19@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10411.383464] LustreError: 325412:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 15 previous similar messages [10413.365586] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10413.469363] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10416.469137] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10418.684408] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10418.719456] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [10418.799891] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10419.001121] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [10428.039451] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10430.008531] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10434.664209] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [10436.604648] Lustre: Failing over lustre-MDT0000 [10436.724296] LustreError: 328935:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [10436.726022] LustreError: 328935:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [10436.787113] Lustre: server umount lustre-MDT0000 complete [10439.141969] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10439.162186] Lustre: Skipped 3 previous similar messages [10439.169034] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10444.268129] LustreError: 327575:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10444.281045] LustreError: 327575:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 10 previous similar messages [10455.519113] Lustre: 319288:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1768556976/real 1768556976] req@0000000089c8667e x1854466097766272/t0(0) o400->MGC192.168.202.119@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1768556992 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10455.535493] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10456.166974] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10465.014452] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10465.070949] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10468.698851] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10468.928901] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [10470.382506] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10470.387935] Lustre: Skipped 2 previous similar messages [10470.402082] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [10470.441859] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [10475.886838] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10477.491301] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10485.729639] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10485.738553] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10485.753532] Lustre: Skipped 1 previous similar message [10485.770930] Lustre: Skipped 2 previous similar messages [10488.690705] LustreError: 329968:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [10488.699373] LustreError: 329968:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [10488.959649] Lustre: server umount lustre-MDT0000 complete [10492.280300] LustreError: 325397:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768557029 with bad export cookie 14321734067691881655 [10492.289261] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10492.301611] LustreError: 325397:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [10492.635334] Lustre: server umount lustre-MDT0001 complete [10506.117768] LustreError: 330370:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [10506.126115] LustreError: 330370:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [10506.283557] Lustre: server umount lustre-OST0000 complete [10520.660364] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10521.209026] LustreError: 331197:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10521.230662] LustreError: 331197:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 23 previous similar messages [10525.425818] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [10532.913046] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10533.217801] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10533.222282] Lustre: Skipped 1 previous similar message [10537.461884] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [10543.435677] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10547.163706] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10554.423929] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10561.269251] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [10566.066485] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [10566.942631] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10567.468785] Lustre: lustre-MDT0001: Client a042bc2b-5b9d-4681-a760-e2650f584074 (at 192.168.202.19@tcp) reconnecting [10571.239988] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10571.243234] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10571.254893] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10571.260774] Lustre: Skipped 2 previous similar messages [10572.927110] LustreError: 333438:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [10572.934734] LustreError: 333438:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [10573.053845] Lustre: server umount lustre-OST0000 complete [10579.429147] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10579.439544] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10579.444817] Lustre: Skipped 1 previous similar message [10583.023198] Lustre: server umount lustre-MDT0000 complete [10586.488188] LustreError: 331176:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768557123 with bad export cookie 14321734067691882383 [10586.495580] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10586.506152] LustreError: 331176:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [10586.943901] Lustre: server umount lustre-MDT0001 complete [10598.465348] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [10601.720563] Key type lgssc unregistered [10602.064311] LNet: 334373:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10602.074790] LNetError: 334373:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [10602.089858] LNet: Removed LNI 192.168.202.119@tcp [10603.284562] Key type .llcrypt unregistered [10603.290122] Key type ._llcrypt unregistered [10630.575449] Key type ._llcrypt registered [10630.577162] Key type .llcrypt registered [10630.693609] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [10631.891968] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10631.967371] alg: No test for adler32 (adler32-zlib) [10633.174755] Lustre: Lustre: Build Version: 2.17.50_30_gbb2e37c [10633.407592] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [10635.095219] Key type lgssc registered [10636.212944] Lustre: Echo OBD driver; http://www.lustre.org/ [10643.858858] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10643.874210] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10645.254471] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10649.721585] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [10654.696527] LustreError: 335871:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10654.724700] LustreError: 335871:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10659.087269] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10659.531975] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10663.767945] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [10668.712555] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10672.763224] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10679.877552] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10680.295307] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10686.444482] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [10690.544465] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [10692.459488] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10705.898703] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10705.900698] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10705.905514] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10705.981763] Lustre: Skipped 2 previous similar messages [10707.713256] LustreError: 338114:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [10708.182315] Lustre: server umount lustre-MDT0000 complete [10711.009447] LustreError: 335866:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10711.042351] LustreError: 335866:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10713.046290] LustreError: 335853:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768557250 with bad export cookie 3510321544360952706 [10713.052728] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10713.060637] LustreError: 335853:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [10713.178627] LustreError: 338316:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [10713.183426] LustreError: 338316:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [10713.350219] Lustre: server umount lustre-MDT0001 complete [10726.674796] LustreError: 338516:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [10726.677183] LustreError: 338516:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [10726.839931] Lustre: server umount lustre-OST0000 complete [10741.859863] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10742.556263] LustreError: 339247:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10742.683351] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10747.616925] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [10747.873653] LustreError: 339247:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10752.994448] LustreError: 339247:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10756.026790] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10756.267796] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10762.106913] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [10771.374487] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10776.816343] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10783.597277] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10783.804689] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10787.882089] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [10789.491210] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [10796.849841] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10811.380282] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10811.383088] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10811.403703] Lustre: Skipped 2 previous similar messages [10811.415978] Lustre: Skipped 4 previous similar messages [10815.895892] LustreError: 341593:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [10815.901486] LustreError: 341593:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [10816.285490] Lustre: server umount lustre-MDT0000 complete [10816.480613] LustreError: 339247:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10816.489938] LustreError: 339247:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10820.393568] LustreError: 339227:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768557357 with bad export cookie 3510321544360953819 [10820.394992] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10820.402409] LustreError: 339227:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [10820.584729] LustreError: 341796:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [10820.594766] LustreError: 341796:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [10820.804314] Lustre: server umount lustre-MDT0001 complete [10835.479520] LustreError: 341997:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [10835.490497] LustreError: 341997:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [10835.650534] Lustre: server umount lustre-OST0000 complete [10851.098053] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10851.558422] LustreError: 342825:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10851.621364] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10855.827270] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [10865.321282] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10870.374531] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [10876.425459] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10881.271869] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10889.109563] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10889.707608] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10889.709943] Lustre: Skipped 1 previous similar message [10896.911161] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [10897.476193] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [10905.328720] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10911.201683] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10911.216606] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10911.229543] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10912.226448] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10912.294464] LustreError: 345069:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [10912.298260] LustreError: 345069:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [10912.367442] Lustre: server umount lustre-OST0000 complete [10917.344427] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10917.354093] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10917.359065] Lustre: Skipped 2 previous similar messages [10917.369069] Lustre: Skipped 3 previous similar messages [10922.467809] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10922.470625] Lustre: Skipped 1 previous similar message [10923.680254] Lustre: server umount lustre-MDT0000 complete [10927.593788] LustreError: 342824:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10927.618667] LustreError: 342824:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10928.254357] LustreError: 342806:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768557465 with bad export cookie 3510321544360954841 [10928.266286] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10928.917108] Lustre: server umount lustre-MDT0001 complete [10940.791451] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [10943.681535] Key type lgssc unregistered [10943.932835] LNet: 346004:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10943.936855] LNetError: 346004:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [10943.950946] LNet: Removed LNI 192.168.202.119@tcp [10944.805313] Key type .llcrypt unregistered [10944.807741] Key type ._llcrypt unregistered [10969.665646] Key type ._llcrypt registered [10969.670592] Key type .llcrypt registered [10969.813953] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [10971.546598] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10971.573029] alg: No test for adler32 (adler32-zlib) [10972.853897] Lustre: Lustre: Build Version: 2.17.50_30_gbb2e37c [10973.120939] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [10974.879346] Key type lgssc registered [10976.363256] Lustre: Echo OBD driver; http://www.lustre.org/ [10985.022873] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10985.038821] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10985.047590] LustreError: 347426:0:(osd_handler.c:8561:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [10985.075175] LustreError: 347426:0:(obd_config.c:783:class_setup()) setup lustre-MDT0000-osd failed (-22) [10985.092451] LustreError: 347426:0:(obd_mount.c:193:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [10985.103280] LustreError: 347426:0:(tgt_mount.c:2451:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [10985.121180] LustreError: 347426:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -22 [10995.010107] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: user_xattr,acl,no_mbcache,nodelalloc [10995.017410] LustreError: 347823:0:(osd_handler.c:8561:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [10995.033168] LustreError: 347823:0:(obd_config.c:783:class_setup()) setup lustre-OST0000-osd failed (-22) [10995.037984] LustreError: 347823:0:(obd_mount.c:193:lustre_start_simple()) lustre-OST0000-osd setup error -22 [10995.043515] LustreError: 347823:0:(tgt_mount.c:2451:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [10995.049339] LustreError: 347823:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -22 [11018.035318] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [11021.356733] Key type lgssc unregistered [11021.634499] LNet: 348810:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11021.650743] LNetError: 348810:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [11021.668755] LNet: Removed LNI 192.168.202.119@tcp [11022.790686] Key type .llcrypt unregistered [11022.792335] Key type ._llcrypt unregistered [11045.865413] Key type ._llcrypt registered [11045.871642] Key type .llcrypt registered [11046.046230] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [11062.152757] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [11063.620932] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11063.729608] alg: No test for adler32 (adler32-zlib) [11065.061859] Lustre: Lustre: Build Version: 2.17.50_30_gbb2e37c [11065.496639] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [11067.223185] Key type lgssc registered [11068.687196] Lustre: Echo OBD driver; http://www.lustre.org/ [11113.943735] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11113.970034] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11115.416947] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11115.490278] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11115.592335] Lustre: lustre-MDT0000: new disk, initializing [11115.725136] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11115.749062] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11122.342535] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [11136.002804] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11136.144864] Lustre: 351933:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11136.199582] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11136.203556] Lustre: Skipped 1 previous similar message [11136.277333] Lustre: lustre-MDT0001: new disk, initializing [11136.380292] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11136.447696] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11136.467357] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11141.633762] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [11150.951389] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11156.644949] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11164.425326] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11164.814299] Lustre: lustre-OST0000: new disk, initializing [11164.818333] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11164.946535] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11171.443093] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11171.453192] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11171.602043] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11171.878600] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [11181.211384] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11186.832363] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11187.155820] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11192.666486] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11192.938957] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11197.415844] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11197.418493] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11197.421490] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11197.441808] Lustre: Skipped 1 previous similar message [11197.477236] LustreError: Skipped 1 previous similar message [11201.168212] LustreError: 354875:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [11201.501242] Lustre: server umount lustre-OST0000 complete [11207.649689] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11207.669610] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11207.681310] Lustre: Skipped 2 previous similar messages [11212.211085] LustreError: 355080:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [11212.218291] LustreError: 355080:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [11212.444448] Lustre: server umount lustre-MDT0000 complete [11217.454574] LustreError: 351925:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768557754 with bad export cookie 3341622359307084456 [11217.455498] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11217.470141] LustreError: 351925:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [11217.643276] LustreError: 355282:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [11217.649671] LustreError: 355282:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [11218.006303] Lustre: server umount lustre-MDT0001 complete [11228.171676] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [11230.247490] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 05:02:45 (1768557765) [11238.274832] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11238.712528] LustreError: 356001:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11238.729470] LustreError: 356001:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [11238.811444] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11243.726426] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [11244.003664] LustreError: 356001:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11249.123196] LustreError: 356001:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11254.109497] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11254.241053] LustreError: 356001:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11254.636845] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11260.919240] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [11267.445151] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11271.467114] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11278.688086] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11278.952556] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11285.765314] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [11292.063593] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11299.039671] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11299.187711] Lustre: lustre-OST0001: new disk, initializing [11299.190918] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11299.277278] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11305.241416] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [11306.605634] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11306.615977] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11306.712430] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11313.047208] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [11317.216461] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11317.229957] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11317.235338] LustreError: Skipped 1 previous similar message [11317.237212] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11317.271421] Lustre: Skipped 2 previous similar messages [11320.170098] Lustre: lustre-OST0001: Not available for connect from 192.168.202.19@tcp (stopping) [11320.182119] Lustre: Skipped 1 previous similar message [11323.034224] LustreError: 359195:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [11323.039832] LustreError: 359195:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [11323.105664] Lustre: server umount lustre-OST0001 complete [11325.293606] LustreError: 357876:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.19@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11337.705113] LustreError: 357597:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11337.736914] LustreError: 357597:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [11341.807734] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11341.816223] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11341.828680] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11341.832369] Lustre: Skipped 2 previous similar messages [11347.591115] LustreError: 359421:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [11347.595676] LustreError: 359421:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [11347.736730] Lustre: server umount lustre-OST0000 complete [11352.033823] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11352.047675] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11352.068656] Lustre: Skipped 1 previous similar message [11352.076633] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11352.081740] Lustre: Skipped 2 previous similar messages [11357.953930] LustreError: 359622:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [11357.957306] LustreError: 359622:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [11358.181451] LustreError: 356000:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11358.199752] LustreError: 356000:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [11358.309481] Lustre: server umount lustre-MDT0000 complete [11363.232386] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11363.230232] LustreError: 355981:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768557900 with bad export cookie 3341622359307085338 [11363.252769] LustreError: 355981:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [11369.755144] Lustre: server umount lustre-MDT0001 complete [11382.351361] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [11385.334432] Key type lgssc unregistered [11385.602943] LNet: 360359:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11385.615169] LNetError: 360359:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [11386.671696] LNet: Removed LNI 192.168.202.119@tcp [11387.561562] Key type .llcrypt unregistered [11387.565606] Key type ._llcrypt unregistered [11445.301508] Key type ._llcrypt registered [11445.306858] Key type .llcrypt registered [11445.492731] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [11447.073280] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11447.128411] alg: No test for adler32 (adler32-zlib) [11448.398635] Lustre: Lustre: Build Version: 2.17.50_30_gbb2e37c [11448.790092] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [11450.503575] Key type lgssc registered [11451.414982] Lustre: Echo OBD driver; http://www.lustre.org/ [11459.685307] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11459.711517] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11460.974767] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [11460.994677] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11461.129326] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11465.806575] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [11475.108926] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11475.212220] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11475.254130] Lustre: 363141:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11475.594891] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11480.299151] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [11487.495085] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11492.148373] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11495.394466] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11495.419716] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11495.425668] Lustre: Skipped 1 previous similar message [11500.518086] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11500.523230] Lustre: Skipped 1 previous similar message [11500.602189] LustreError: 364507:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [11500.716676] Lustre: server umount lustre-MDT0000 complete [11504.794209] LustreError: 363131:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768558042 with bad export cookie 17948652310864965182 [11504.797748] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11504.806117] LustreError: 363131:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [11505.010180] LustreError: 364708:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [11505.017486] LustreError: 364708:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [11505.402662] Lustre: server umount lustre-MDT0001 complete [11519.513739] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 05:07:34 (1768558054) [11527.580310] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11528.318834] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11533.699396] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [11533.797033] LustreError: 365428:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11533.816766] LustreError: 365428:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11538.915458] LustreError: 365428:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11543.484611] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11543.787257] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11548.313263] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [11555.022088] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11559.246784] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11566.450366] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11566.729735] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [11566.965092] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11573.213264] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [11579.127843] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11586.214460] Lustre: Permanently deactivating lustre-OST0000 [11586.228420] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [11591.733783] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11593.906562] LustreError: 368026:0:(mgs_llog.c:1942:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [11593.914515] LustreError: 368026:0:(mgs_llog.c:1997:mgs_replace_nids()) Only MGS is allowed to be started [11593.921094] LustreError: 368026:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11597.773453] LustreError: 368074:0:(mgs_llog.c:1997:mgs_replace_nids()) Only MGS is allowed to be started [11597.780513] LustreError: 368074:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11605.985105] LustreError: 368172:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [11605.986897] LustreError: 368172:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [11606.110236] Lustre: server umount lustre-OST0000 complete [11609.244327] LustreError: 368325:0:(mgs_llog.c:1997:mgs_replace_nids()) Only MGS is allowed to be started [11609.252122] LustreError: 368325:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11611.622363] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11611.631096] Lustre: Skipped 1 previous similar message [11611.638205] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11611.651266] Lustre: Skipped 1 previous similar message [11616.739741] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11616.753721] Lustre: Skipped 1 previous similar message [11620.860161] LustreError: 368422:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [11620.874257] LustreError: 368422:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [11620.974891] Lustre: server umount lustre-MDT0000 complete [11621.858085] LustreError: 365428:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11621.888478] LustreError: 365428:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11625.421420] LustreError: 365408:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768558162 with bad export cookie 17948652310864965763 [11625.427610] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11625.438821] LustreError: 365408:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [11634.688492] LustreError: 368624:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [11634.710605] LustreError: 368624:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [11634.887553] Lustre: server umount lustre-MDT0001 complete [11647.719410] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11653.594601] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [11656.834685] LustreError: 369467:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [11664.499126] Lustre: 369807:0:(mgs_llog.c:1709:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.119@tcp [11666.653831] LustreError: 369904:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [11666.662450] LustreError: 369904:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [11666.743876] Lustre: server umount MGS complete [11679.516250] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11680.233271] LustreError: 370532:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11680.266085] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11680.272560] Lustre: Skipped 1 previous similar message [11680.317222] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11685.350264] LustreError: 370532:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11686.723593] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [11695.589574] LustreError: 370532:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11695.629471] LustreError: 370532:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11698.488689] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11699.147605] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11699.259172] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11703.287611] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [11708.881591] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11712.762524] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11714.714583] Lustre: Permanently reactivating lustre-OST0000 [11714.763319] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [11714.766673] Lustre: Skipped 2 previous similar messages [11744.228020] LustreError: 372123:0:(osp_dev.c:692:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [11752.585843] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11752.946905] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11760.198402] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [11766.333352] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11772.313180] LustreError: 373703:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [11772.317447] LustreError: 373703:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [11772.478432] Lustre: server umount lustre-OST0000 complete [11779.559868] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11779.574055] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11779.593955] Lustre: Skipped 2 previous similar messages [11779.610752] Lustre: Skipped 1 previous similar message [11785.810062] Lustre: server umount lustre-MDT0000 complete [11789.794778] Lustre: 362465:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1768558311/real 1768558311] req@000000007a047692 x1854467458822784/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1768558327 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11789.798770] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11789.799332] LustreError: 370526:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11789.821442] Lustre: 362465:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 1 previous similar message [11790.534165] LustreError: 371879:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768558327 with bad export cookie 17948652310864966813 [11790.536766] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11790.550403] LustreError: 371879:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [11791.232950] Lustre: server umount lustre-MDT0001 complete [11802.683382] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [11804.772702] Key type lgssc unregistered [11805.049189] LNet: 374640:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11805.055322] LNetError: 374640:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [11805.068609] LNet: Removed LNI 192.168.202.119@tcp [11805.599760] Key type .llcrypt unregistered [11805.601110] Key type ._llcrypt unregistered [11821.174179] Key type ._llcrypt registered [11821.176229] Key type .llcrypt registered [11821.264736] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [11835.128321] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [11836.322262] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11836.376862] alg: No test for adler32 (adler32-zlib) [11837.442143] Lustre: Lustre: Build Version: 2.17.50_30_gbb2e37c [11837.643667] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [11839.327214] Key type lgssc registered [11840.476165] Lustre: Echo OBD driver; http://www.lustre.org/ [11873.512294] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11873.565761] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11874.919873] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11874.952891] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11875.017227] Lustre: lustre-MDT0000: new disk, initializing [11875.098623] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11875.119590] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11879.743662] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [11890.838844] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11891.010492] Lustre: 377761:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11891.091082] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11891.100684] Lustre: Skipped 1 previous similar message [11891.236193] Lustre: lustre-MDT0001: new disk, initializing [11891.372842] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11891.411944] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11891.422171] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11896.274197] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [11904.786742] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11908.969439] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11915.633703] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11915.942808] Lustre: lustre-OST0000: new disk, initializing [11915.946335] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11916.030943] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11923.119263] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [11923.564142] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11923.568916] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11923.731557] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11931.643230] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11939.020645] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11939.422444] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11944.431837] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11944.724540] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11949.540990] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11949.546730] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11949.559578] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11954.661883] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11954.674303] Lustre: Skipped 1 previous similar message [11959.777879] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11959.781706] Lustre: Skipped 1 previous similar message [11961.311186] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11961.368888] LustreError: 380697:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [11961.559397] Lustre: server umount lustre-OST0000 complete [11968.480609] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11968.493265] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11968.507678] Lustre: Skipped 1 previous similar message [11968.519176] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11968.521612] Lustre: Skipped 1 previous similar message [11970.016865] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11971.935258] LustreError: 380900:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [11971.948247] LustreError: 380900:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [11972.120104] Lustre: server umount lustre-MDT0000 complete [11976.383970] LustreError: 379506:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768558513 with bad export cookie 14222138953509948114 [11976.397046] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11976.399230] LustreError: 379506:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [11976.675925] LustreError: 381101:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [11976.691741] LustreError: 381101:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [11977.144668] Lustre: server umount lustre-MDT0001 complete [11987.400249] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [11989.352361] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 05:15:25 (1768558525) [11997.593632] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11998.099717] LustreError: 381820:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11998.118961] LustreError: 381820:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [11998.171060] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12001.865172] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [12003.303191] LustreError: 381820:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12008.437748] LustreError: 381820:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12012.527127] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12012.820169] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12017.651410] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [12025.014606] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12030.115633] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12038.146259] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12038.756964] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12049.016314] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [12058.251419] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12061.234903] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [12063.949875] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [12073.953913] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12073.955046] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12073.958072] LustreError: Skipped 1 previous similar message [12073.958616] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12073.958619] Lustre: Skipped 2 previous similar messages [12073.970352] Lustre: Skipped 1 previous similar message [12076.671203] LustreError: 384257:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [12076.673849] LustreError: 384257:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [12076.857724] Lustre: server umount lustre-OST0000 complete [12084.196095] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12084.200666] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12084.203319] Lustre: Skipped 1 previous similar message [12084.207916] Lustre: Skipped 2 previous similar messages [12087.367782] LustreError: 384459:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [12087.378591] LustreError: 384459:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [12087.478791] Lustre: server umount lustre-MDT0000 complete [12089.326601] LustreError: 383205:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12089.359493] LustreError: 383205:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12093.051038] LustreError: 382537:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768558630 with bad export cookie 14222138953509948989 [12093.053337] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12093.063927] LustreError: 382537:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [12093.648986] Lustre: server umount lustre-MDT0001 complete [12108.007709] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [12112.187567] Key type lgssc unregistered [12112.425173] LNet: 385193:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12112.436492] LNetError: 385193:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [12112.456251] LNet: Removed LNI 192.168.202.119@tcp [12113.797815] Key type .llcrypt unregistered [12113.800775] Key type ._llcrypt unregistered [12141.438736] Key type ._llcrypt registered [12141.442555] Key type .llcrypt registered [12141.575893] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [12141.902404] Key type .llcrypt unregistered [12141.904101] Key type ._llcrypt unregistered [12161.349400] Key type ._llcrypt registered [12161.351633] Key type .llcrypt registered [12161.532854] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [12162.819441] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12162.854823] alg: No test for adler32 (adler32-zlib) [12164.003975] Lustre: Lustre: Build Version: 2.17.50_30_gbb2e37c [12164.263119] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [12165.991573] Key type lgssc registered [12167.316692] Lustre: Echo OBD driver; http://www.lustre.org/ [12176.418333] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12176.437114] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12178.068623] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12183.382670] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [12187.622593] LustreError: 387366:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12187.648867] LustreError: 387366:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12192.740125] LustreError: 387366:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12195.734872] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12196.074878] LustreError: 387366:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12196.145524] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12201.449289] LustreError: 387362:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12201.484151] LustreError: 387362:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12203.525394] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [12203.668839] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [12208.614676] LustreError: 387366:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12208.653721] LustreError: 387366:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [12210.061679] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12218.138660] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12218.525268] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12223.302196] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [12233.530115] LustreError: 389418:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [12233.705073] Lustre: server umount lustre-OST0000 complete [12238.817601] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12238.828389] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12238.850109] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12239.335235] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12244.448978] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12244.460339] Lustre: Skipped 3 previous similar messages [12247.537495] LustreError: 389619:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [12247.544687] LustreError: 389619:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [12247.621783] Lustre: server umount lustre-MDT0000 complete [12249.567332] Lustre: 386679:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1768558771/real 1768558771] req@00000000f5317c5a x1854468208486272/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1768558787 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12249.573903] LustreError: 387366:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12249.609759] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12249.647577] LustreError: 387366:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [12250.593073] Lustre: 386677:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1768558771/real 1768558771] req@00000000d1d187d1 x1854468208487168/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1768558787 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12252.094696] LustreError: 387347:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768558789 with bad export cookie 4571908251022237578 [12252.107336] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12252.224874] LustreError: 389821:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [12252.233984] LustreError: 389821:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [12252.460067] Lustre: server umount lustre-MDT0001 complete [12265.995283] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [12269.398625] Key type lgssc unregistered [12269.857284] LNet: 390354:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12269.866835] LNetError: 390354:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [12269.881975] LNet: Removed LNI 192.168.202.119@tcp [12271.088577] Key type .llcrypt unregistered [12271.092093] Key type ._llcrypt unregistered [12305.809977] Key type ._llcrypt registered [12305.819786] Key type .llcrypt registered [12306.029962] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [12307.753345] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12307.867449] alg: No test for adler32 (adler32-zlib) [12309.128610] Lustre: Lustre: Build Version: 2.17.50_30_gbb2e37c [12309.527093] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [12311.295265] Key type lgssc registered [12312.785678] Lustre: Echo OBD driver; http://www.lustre.org/ [12322.237962] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12322.258196] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12337.887720] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [12343.007236] LustreError: 391725:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.119@tcp: failed processing log, type 4: rc = -110 [12369.823621] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12377.114553] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [12384.354454] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12392.757928] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12393.254986] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12393.383098] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [12398.048185] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [12398.566240] LustreError: 391751:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12398.598255] LustreError: 391751:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12403.682203] LustreError: 391751:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12403.707783] LustreError: 391751:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12407.448758] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12407.709123] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12413.047893] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [12421.215603] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12426.657316] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12438.499679] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12438.510352] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12438.511712] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12441.215238] LustreError: 394000:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [12441.344220] Lustre: server umount lustre-OST0000 complete [12448.739560] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12448.739917] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12448.752373] Lustre: Skipped 1 previous similar message [12448.758719] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12448.758974] Lustre: Skipped 1 previous similar message [12451.785713] LustreError: 394202:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [12451.794792] LustreError: 394202:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [12452.117855] Lustre: server umount lustre-MDT0000 complete [12453.867278] LustreError: 392652:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12453.889526] LustreError: 392652:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12455.514169] LustreError: 391731:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768558992 with bad export cookie 7400624416744325435 [12455.521047] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12455.533108] LustreError: 391731:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [12455.697856] LustreError: 394403:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [12455.700612] LustreError: 394403:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [12455.918690] Lustre: server umount lustre-MDT0001 complete [12467.715771] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [12470.793259] Key type lgssc unregistered [12471.140791] LNet: 394936:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12471.146895] LNetError: 394936:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [12471.182493] LNet: Removed LNI 192.168.202.119@tcp [12472.096853] Key type .llcrypt unregistered [12472.100476] Key type ._llcrypt unregistered [12499.897514] Key type ._llcrypt registered [12499.901186] Key type .llcrypt registered [12500.055561] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [12501.214791] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12501.276384] alg: No test for adler32 (adler32-zlib) [12502.718534] Lustre: Lustre: Build Version: 2.17.50_30_gbb2e37c [12503.042436] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [12504.831216] Key type lgssc registered [12506.921740] Lustre: Echo OBD driver; http://www.lustre.org/ [12516.710336] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12516.744439] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12518.500593] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12524.013666] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [12527.594701] LustreError: 396335:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12527.610283] LustreError: 396335:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12532.709899] LustreError: 396335:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12534.798303] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12535.298080] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12541.121726] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [12549.375991] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12554.683494] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12562.106084] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12562.508562] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12563.531162] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [12570.464591] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [12578.176851] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12584.945791] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12584.959820] Lustre: Skipped 2 previous similar messages [12584.960724] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12584.971691] Lustre: Skipped 1 previous similar message [12586.361512] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.19@tcp (stopping) [12589.870402] LustreError: 398578:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [12590.061722] LustreError: 396331:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12590.080459] Lustre: server umount lustre-MDT0000 complete [12590.087540] LustreError: 396331:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12593.002408] LustreError: 396315:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.19@tcp arrived at 1768559130 with bad export cookie 6710390488285023152 [12595.178788] LustreError: 396336:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12595.218573] LustreError: 396336:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [12595.326628] LustreError: 396315:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768559132 with bad export cookie 6710390488285022263 [12595.340639] LustreError: 396315:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [12595.342204] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12600.289497] LustreError: 396336:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12600.290210] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12600.290353] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12600.294674] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12600.314702] LustreError: 396336:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12601.474400] LustreError: 398779:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [12601.479932] LustreError: 398779:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [12601.926364] Lustre: server umount lustre-OST0000 complete [12610.277199] LustreError: 399136:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [12610.292621] LustreError: 399136:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [12610.505518] Lustre: server umount lustre-MDT0001 complete [12622.897088] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [12626.473284] Key type lgssc unregistered [12626.827539] LNet: 399669:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12626.830403] LNetError: 399669:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [12626.847853] LNet: Removed LNI 192.168.202.119@tcp [12627.977659] Key type .llcrypt unregistered [12627.980410] Key type ._llcrypt unregistered [12658.549497] Key type ._llcrypt registered [12658.552125] Key type .llcrypt registered [12658.725987] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [12659.835296] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12659.886594] alg: No test for adler32 (adler32-zlib) [12661.265263] Lustre: Lustre: Build Version: 2.17.50_30_gbb2e37c [12661.626408] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [12663.359418] Key type lgssc registered [12664.709725] Lustre: Echo OBD driver; http://www.lustre.org/ [12675.476265] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12675.515609] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12677.263621] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12681.875826] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [12686.819786] LustreError: 401065:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12686.856630] LustreError: 401065:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12691.799978] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12691.938890] LustreError: 401065:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12692.404717] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12697.714489] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [12706.063678] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12710.453680] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12719.018072] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12719.598348] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12721.659347] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [12727.345137] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [12734.466561] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12742.114849] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12742.133566] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12746.053280] LustreError: 403311:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [12746.219697] Lustre: server umount lustre-MDT0001 complete [12747.237991] LustreError: 401060:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12747.277811] LustreError: 401060:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12752.355988] LustreError: 401061:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12752.398504] LustreError: 401061:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12757.474753] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12757.479099] LustreError: 401065:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12757.497065] Lustre: Skipped 1 previous similar message [12757.511271] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12757.548169] Lustre: Skipped 2 previous similar messages [12760.788464] LustreError: 403513:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [12760.792834] LustreError: 403513:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [12760.938425] Lustre: server umount lustre-OST0000 complete [12765.547559] LustreError: 403714:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [12765.556740] LustreError: 403714:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [12765.749293] Lustre: server umount lustre-MDT0000 complete [12782.125170] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [12785.314868] Key type lgssc unregistered [12785.623666] LNet: 404399:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12785.629575] LNetError: 404399:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [12785.644845] LNet: Removed LNI 192.168.202.119@tcp [12786.648701] Key type .llcrypt unregistered [12786.654487] Key type ._llcrypt unregistered [12815.459965] Key type ._llcrypt registered [12815.466408] Key type .llcrypt registered [12815.599521] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [12830.843240] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [12832.601052] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12832.650425] alg: No test for adler32 (adler32-zlib) [12833.764577] Lustre: Lustre: Build Version: 2.17.50_30_gbb2e37c [12834.026095] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [12835.711197] Key type lgssc registered [12837.146603] Lustre: Echo OBD driver; http://www.lustre.org/ [12881.364280] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [12898.254174] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12898.286419] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12899.681754] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12899.702497] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12899.770620] Lustre: lustre-MDT0000: new disk, initializing [12899.861965] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12899.899110] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12905.275749] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [12922.264074] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12922.604425] Lustre: lustre-OST0000: new disk, initializing [12922.617317] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12922.620802] Lustre: Skipped 1 previous similar message [12922.710666] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12929.923319] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [12931.725313] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [12931.742370] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [12931.777604] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [12938.132406] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12954.123424] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12954.219623] Lustre: 408364:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12954.290277] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12954.368411] Lustre: lustre-MDT0001: new disk, initializing [12954.518533] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12954.555049] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [12954.562711] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [12959.389572] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [12970.773042] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [12971.096947] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [12976.109182] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [12976.373411] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [12984.288849] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12984.294941] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12984.304900] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12984.338683] Lustre: Skipped 1 previous similar message [12986.220208] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.19@tcp (stopping) [12986.223067] Lustre: Skipped 1 previous similar message [12987.829196] LustreError: 411773:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [12987.953031] Lustre: server umount lustre-MDT0001 complete [12994.540419] LustreError: 408369:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12994.565589] LustreError: 408369:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [12999.654119] LustreError: 408369:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12999.678607] LustreError: 408369:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [13004.773946] LustreError: 408371:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13004.792750] LustreError: 408371:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [13008.864501] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13008.870902] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13008.888475] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13009.889163] LustreError: 408374:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13012.094511] LustreError: 412045:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [13012.099212] LustreError: 412045:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [13012.317671] Lustre: server umount lustre-OST0000 complete [13015.015145] LustreError: 408374:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13022.550241] LustreError: 412246:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [13022.562533] LustreError: 412246:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [13022.809807] Lustre: server umount lustre-MDT0000 complete [13037.225983] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [13040.216163] Key type lgssc unregistered [13040.576793] LNet: 412932:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13040.590220] LNetError: 412932:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [13040.601984] LNet: Removed LNI 192.168.202.119@tcp [13041.302623] Key type .llcrypt unregistered [13041.305856] Key type ._llcrypt unregistered [13086.645649] Key type ._llcrypt registered [13086.648574] Key type .llcrypt registered [13086.762973] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [13086.988760] Key type .llcrypt unregistered [13086.991856] Key type ._llcrypt unregistered [13105.853162] Key type ._llcrypt registered [13105.855176] Key type .llcrypt registered [13106.019054] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [13107.265986] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13107.551328] alg: No test for adler32 (adler32-zlib) [13108.834511] Lustre: Lustre: Build Version: 2.17.50_30_gbb2e37c [13109.054707] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [13110.792235] Key type lgssc registered [13112.497138] Lustre: Echo OBD driver; http://www.lustre.org/ [13145.020610] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13145.066723] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13146.527747] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13146.560308] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13146.686669] Lustre: lustre-MDT0000: new disk, initializing [13146.867289] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13146.921421] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13152.684145] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [13164.167835] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13164.312725] Lustre: 416272:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13164.392493] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13164.405782] Lustre: Skipped 1 previous similar message [13164.538268] Lustre: lustre-MDT0001: new disk, initializing [13164.704676] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13164.752451] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13164.769714] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13169.972775] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [13179.565924] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13184.762986] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13193.725698] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13194.086539] Lustre: lustre-OST0000: new disk, initializing [13194.092330] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13194.211320] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13199.439515] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13199.447978] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13199.556134] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13201.177667] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [13210.432652] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13220.330987] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13220.335513] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13220.347366] Lustre: Skipped 1 previous similar message [13220.365155] Lustre: Skipped 2 previous similar messages [13223.355759] LustreError: 418709:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [13223.455619] Lustre: server umount lustre-MDT0000 complete [13228.498404] LustreError: 416264:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768559765 with bad export cookie 18145123640128891207 [13228.512612] LustreError: 416264:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [13228.516662] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13228.743309] LustreError: 418911:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [13228.753537] LustreError: 418911:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [13229.055245] Lustre: server umount lustre-MDT0001 complete [13244.040382] LustreError: 419112:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [13244.044705] LustreError: 419112:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [13244.221417] Lustre: server umount lustre-OST0000 complete [13257.377826] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 05:36:32 (1768559792) [13267.769574] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13268.507332] LustreError: 419932:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13268.527408] LustreError: 419932:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [13268.605444] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13273.573668] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [13274.084156] LustreError: 419932:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13278.177755] LustreError: 419932:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13283.271471] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13283.298984] LustreError: 419932:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13283.623372] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13288.229316] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [13295.505699] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13300.105964] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13319.594899] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13319.792344] Lustre: Found index 0 for lustre-OST0000, updating log [13320.021168] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13326.651328] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [13333.084785] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13338.480272] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [13348.835811] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13348.836963] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13348.846771] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13348.892552] Lustre: Skipped 2 previous similar messages [13349.924179] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13349.926449] Lustre: Skipped 1 previous similar message [13350.728644] LustreError: 422188:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [13350.742073] LustreError: 422188:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [13350.983329] Lustre: server umount lustre-OST0000 complete [13370.848540] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13370.854101] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13370.864697] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13370.867172] Lustre: Skipped 1 previous similar message [13373.921074] LustreError: 422443:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [13373.937477] LustreError: 422443:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [13374.224341] Lustre: server umount lustre-MDT0000 complete [13376.482936] LustreError: 419932:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13376.525760] LustreError: 419932:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [13379.125517] LustreError: 419914:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768559916 with bad export cookie 18145123640128892600 [13379.132788] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13379.148036] LustreError: 419914:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [13379.722981] Lustre: server umount lustre-MDT0001 complete [13385.695976] LNet: 1 local NIs in recovery (showing 1): 192.168.202.119@tcp [13390.888576] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 05:38:46 (1768559926) [14018.433916] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14019.048066] LustreError: 426010:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14019.084923] LustreError: 426010:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [14019.237831] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14024.641886] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [14024.673271] LustreError: 426010:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14028.771356] LustreError: 426010:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14034.911955] LustreError: 426010:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14034.930859] LustreError: 426010:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [14035.299322] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [14035.483503] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14040.463776] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [14046.550161] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [14051.033753] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [14070.765978] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [14071.025865] Lustre: Found index 0 for lustre-OST0000, updating log [14071.390411] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14077.925020] LNet: 1 local NIs in recovery (showing 1): 192.168.202.119@tcp [14078.942905] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [14084.697822] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [14087.139802] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14087.143681] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14087.152388] Lustre: Skipped 1 previous similar message [14087.164147] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14087.171672] Lustre: Skipped 2 previous similar messages [14088.734792] LustreError: 428276:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [14088.738924] LustreError: 428276:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [14088.940507] Lustre: server umount lustre-OST0000 complete [14094.370474] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14094.386530] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14094.401069] Lustre: Skipped 1 previous similar message [14095.392504] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14097.441752] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14097.449038] Lustre: Skipped 3 previous similar messages [14099.539667] LustreError: 428477:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [14099.544688] LustreError: 428477:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [14099.659867] Lustre: server umount lustre-MDT0000 complete [14100.517025] LustreError: 426005:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14100.530976] LustreError: 426005:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [14103.194584] LustreError: 425989:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768560640 with bad export cookie 18145123640128893678 [14103.195547] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14103.213353] LustreError: 425989:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [14103.303414] LustreError: 428679:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [14103.309627] LustreError: 428679:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [14103.508558] Lustre: server umount lustre-MDT0001 complete [14134.958773] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [14141.407623] LNet: 1 local NIs in recovery (showing 1): 192.168.202.119@tcp [14144.525360] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [14186.371798] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 05:52:02 (1768560722) [14205.074245] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [14208.155211] Key type lgssc unregistered [14208.405785] LNet: 432237:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14208.413292] LNetError: 432237:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [14333.353089] LNet: Removed LNI 192.168.202.119@tcp [14334.225723] Key type .llcrypt unregistered [14334.228518] Key type ._llcrypt unregistered [14350.144458] Key type ._llcrypt registered [14350.149229] Key type .llcrypt registered [14350.258496] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [14351.332668] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14351.358541] alg: No test for adler32 (adler32-zlib) [14352.461120] Lustre: Lustre: Build Version: 2.17.50_30_gbb2e37c [14352.701975] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [14354.383217] Key type lgssc registered [14355.421747] Lustre: Echo OBD driver; http://www.lustre.org/ [14359.521672] LNet: Added LNI 192.168.202.119@tcp100 [8/256/0/180] [14360.392629] LNet: Added LNI 192.168.202.119@tcp101 [8/256/0/180] [14361.277240] LNet: Added LNI 192.168.202.119@tcp102 [8/256/0/180] [14362.240333] LNet: Added LNI 192.168.202.119@tcp103 [8/256/0/180] [14363.082940] LNet: Added LNI 192.168.202.119@tcp104 [8/256/0/180] [14363.906316] LNet: Added LNI 192.168.202.119@tcp105 [8/256/0/180] [14364.747237] LNet: Added LNI 192.168.202.119@tcp106 [8/256/0/180] [14365.634094] LNet: Added LNI 192.168.202.119@tcp107 [8/256/0/180] [14366.511604] LNet: Added LNI 192.168.202.119@tcp108 [8/256/0/180] [14367.338992] LNet: Added LNI 192.168.202.119@tcp109 [8/256/0/180] [14368.415554] LNet: Added LNI 192.168.202.119@tcp110 [8/256/0/180] [14369.244349] LNet: Added LNI 192.168.202.119@tcp111 [8/256/0/180] [14370.060550] LNet: Added LNI 192.168.202.119@tcp112 [8/256/0/180] [14370.949390] LNet: Added LNI 192.168.202.119@tcp113 [8/256/0/180] [14371.904737] LNet: Added LNI 192.168.202.119@tcp114 [8/256/0/180] [14372.693730] LNet: Added LNI 192.168.202.119@tcp115 [8/256/0/180] [14373.502818] LNet: Added LNI 192.168.202.119@tcp116 [8/256/0/180] [14374.139824] LNet: Added LNI 192.168.202.119@tcp117 [8/256/0/180] [14374.963897] LNet: Added LNI 192.168.202.119@tcp118 [8/256/0/180] [14376.007696] LNet: Added LNI 192.168.202.119@tcp119 [8/256/0/180] [14377.288606] LNet: Added LNI 192.168.202.119@tcp120 [8/256/0/180] [14378.540045] LNet: Added LNI 192.168.202.119@tcp121 [8/256/0/180] [14379.505826] LNet: Added LNI 192.168.202.119@tcp122 [8/256/0/180] [14380.910896] LNet: Added LNI 192.168.202.119@tcp123 [8/256/0/180] [14381.850583] LNet: Added LNI 192.168.202.119@tcp124 [8/256/0/180] [14383.072068] LNet: Added LNI 192.168.202.119@tcp125 [8/256/0/180] [14384.326931] LNet: Added LNI 192.168.202.119@tcp126 [8/256/0/180] [14385.195404] LNet: Added LNI 192.168.202.119@tcp127 [8/256/0/180] [14386.198243] LNet: Added LNI 192.168.202.119@tcp128 [8/256/0/180] [14387.415706] LNet: Added LNI 192.168.202.119@tcp129 [8/256/0/180] [14388.465425] LNet: Added LNI 192.168.202.119@tcp130 [8/256/0/180] [14389.497454] LNet: Added LNI 192.168.202.119@tcp131 [8/256/0/180] [14390.452377] LNet: Added LNI 192.168.202.119@tcp132 [8/256/0/180] [14391.488071] LNet: Added LNI 192.168.202.119@tcp133 [8/256/0/180] [14392.406349] LNet: Added LNI 192.168.202.119@tcp134 [8/256/0/180] [14393.577146] LNet: Added LNI 192.168.202.119@tcp135 [8/256/0/180] [14401.856898] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14401.934426] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14403.082818] Lustre: 435252:0:(lustre_peer.c:105:class_add_uuid()) 192.168.202.119@tcp: can't add NID 192.168.202.119@tcp130: rc = -75 [14403.181804] LNetError: 435281:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.119 [14403.205919] LNetError: Protocol error connecting to 192.168.202.119@tcp118 at host 192.168.202.119:988: is it running a compatible version of Lustre? [14403.225882] LNetError: Skipped 2 previous similar messages [14403.375977] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14403.435913] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14403.537703] Lustre: lustre-MDT0000: new disk, initializing [14403.653248] Lustre: 435310:0:(lustre_peer.c:105:class_add_uuid()) 192.168.202.119@tcp: can't add NID 192.168.202.119@tcp130: rc = -75 [14403.669086] Lustre: 435310:0:(lustre_peer.c:105:class_add_uuid()) Skipped 5 previous similar messages [14403.705926] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14403.799031] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14404.000871] LNet: 4 local NIs in recovery (showing 4): 192.168.202.119@tcp126, 192.168.202.119@tcp127, 192.168.202.119@tcp129, 192.168.202.119@tcp128 [14404.256297] LNetError: 432710:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.119 [14404.270558] LNetError: 432710:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 7 previous similar messages [14404.276985] LNetError: Protocol error connecting to 192.168.202.119@tcp103 at host 192.168.202.119:988: is it running a compatible version of Lustre? [14404.282348] LNetError: Skipped 4 previous similar messages [14407.093391] LNetError: 435275:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.119 [14407.110349] LNetError: 435275:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 4 previous similar messages [14407.123248] LNetError: Protocol error connecting to 192.168.202.119@tcp127 at host 192.168.202.119:988: is it running a compatible version of Lustre? [14407.142315] LNetError: Skipped 5 previous similar messages [14407.999850] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [14411.239652] LNetError: 432711:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.119 [14411.259586] LNetError: 432711:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 1 previous similar message [14411.274558] LNetError: Protocol error connecting to 192.168.202.119@tcp127 at host 192.168.202.119:988: is it running a compatible version of Lustre? [14419.065037] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [14419.275441] Lustre: 435269:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14419.389443] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14419.393376] Lustre: Skipped 1 previous similar message [14419.426666] LNetError: 435286:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.119 [14419.450327] LNetError: Protocol error connecting to 192.168.202.119@tcp127 at host 192.168.202.119:988: is it running a compatible version of Lustre? [14419.478542] Lustre: lustre-MDT0001: new disk, initializing [14419.516726] Lustre: 436055:0:(lustre_peer.c:105:class_add_uuid()) 192.168.202.119@tcp: can't add NID 192.168.202.119@tcp130: rc = -75 [14419.526577] Lustre: 436055:0:(lustre_peer.c:105:class_add_uuid()) Skipped 5 previous similar messages [14419.674583] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14419.732315] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14419.775723] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14424.864364] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [14426.161213] Lustre: 436334:0:(lustre_peer.c:105:class_add_uuid()) 192.168.202.119@tcp: can't add NID 192.168.202.119@tcp130: rc = -75 [14426.167541] Lustre: 436334:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [14434.673164] LNetError: 432710:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.119 [14434.692067] LNetError: Protocol error connecting to 192.168.202.119@tcp125 at host 192.168.202.119:988: is it running a compatible version of Lustre? [14435.033987] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [14439.669936] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [14447.846491] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [14448.181846] Lustre: lustre-OST0000: new disk, initializing [14448.185986] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14448.287863] Lustre: 437022:0:(lustre_peer.c:105:class_add_uuid()) 192.168.202.119@tcp: can't add NID 192.168.202.119@tcp130: rc = -75 [14448.293911] Lustre: 437022:0:(lustre_peer.c:105:class_add_uuid()) Skipped 5 previous similar messages [14448.350420] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14451.040498] LNetError: 432711:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.119 [14451.049516] LNetError: 432711:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 8 previous similar messages [14451.058353] LNetError: Protocol error connecting to 192.168.202.119@tcp118 at host 192.168.202.119:988: is it running a compatible version of Lustre? [14451.072183] LNetError: Skipped 8 previous similar messages [14455.699176] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [14457.429844] Lustre: 437431:0:(lustre_peer.c:105:class_add_uuid()) 192.168.202.119@tcp: can't add NID 192.168.202.119@tcp130: rc = -75 [14457.443201] Lustre: 437431:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [14457.544911] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14457.559313] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14457.704699] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14464.415644] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [14464.481578] LNet: 1 local NIs in recovery (showing 1): 192.168.202.119@tcp127 [14473.189912] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14473.190360] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14473.221045] LustreError: Skipped 1 previous similar message [14473.226027] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14478.314092] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14478.319078] Lustre: Skipped 2 previous similar messages [14478.479159] LustreError: 437855:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [14478.844172] Lustre: server umount lustre-OST0000 complete [14486.501417] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14486.510687] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14486.543181] Lustre: Skipped 1 previous similar message [14486.565370] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14488.554121] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14488.582284] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14488.584162] Lustre: Skipped 1 previous similar message [14493.025496] LNetError: 435277:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.119 [14493.032041] LNetError: 435277:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 15 previous similar messages [14493.036975] LNetError: Protocol error connecting to 192.168.202.119@tcp103 at host 192.168.202.119:988: is it running a compatible version of Lustre? [14493.060031] LNetError: Skipped 15 previous similar messages [14493.671087] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14493.678639] Lustre: Skipped 1 previous similar message [14499.296187] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14499.443560] LustreError: 438058:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [14499.451691] LustreError: 438058:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [14499.582455] Lustre: server umount lustre-MDT0000 complete [14503.973089] LustreError: 435262:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768561041 with bad export cookie 10222950833252663675 [14503.980505] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14503.994299] LustreError: 435262:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [14504.191280] LustreError: 438262:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [14504.196835] LustreError: 438262:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [14504.345979] Lustre: server umount lustre-MDT0001 complete [14515.305957] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [14517.961966] Key type lgssc unregistered [14518.398703] LNet: 438802:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14518.406699] LNetError: 438802:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [14519.456068] LNet: Removed LNI 192.168.202.119@tcp [14519.462332] LNetError: 438802:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [14519.480509] LNet: Removed LNI 192.168.202.119@tcp100 [14520.544767] LNet: Removed LNI 192.168.202.119@tcp101 [14520.546945] LNetError: 438802:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [14520.570797] LNetError: 438802:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14520.578875] LNet: Removed LNI 192.168.202.119@tcp102 [14521.635080] LNet: Removed LNI 192.168.202.119@tcp103 [14522.658565] LNet: Removed LNI 192.168.202.119@tcp104 [14522.660671] LNetError: 438802:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [14522.664170] LNetError: 438802:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [14523.679282] LNet: Removed LNI 192.168.202.119@tcp105 [14524.703383] LNet: Removed LNI 192.168.202.119@tcp106 [14525.727428] LNet: Removed LNI 192.168.202.119@tcp107 [14526.751265] LNet: Removed LNI 192.168.202.119@tcp108 [14526.754757] LNetError: 438802:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [14526.765340] LNetError: 438802:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [14526.778685] LNet: Removed LNI 192.168.202.119@tcp109 [14527.839491] LNet: Removed LNI 192.168.202.119@tcp110 [14528.863207] LNet: Removed LNI 192.168.202.119@tcp111 [14529.887224] LNet: Removed LNI 192.168.202.119@tcp112 [14530.915368] LNet: Removed LNI 192.168.202.119@tcp113 [14531.935582] LNet: Removed LNI 192.168.202.119@tcp114 [14532.964614] LNet: Removed LNI 192.168.202.119@tcp115 [14533.983454] LNet: Removed LNI 192.168.202.119@tcp116 [14535.007169] LNet: Removed LNI 192.168.202.119@tcp117 [14535.021437] LNetError: 438802:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [14535.037708] LNetError: 438802:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [14535.054581] LNet: Removed LNI 192.168.202.119@tcp118 [14536.095303] LNet: Removed LNI 192.168.202.119@tcp119 [14537.119455] LNet: Removed LNI 192.168.202.119@tcp120 [14538.148872] LNet: Removed LNI 192.168.202.119@tcp121 [14539.167266] LNet: Removed LNI 192.168.202.119@tcp122 [14540.191337] LNet: Removed LNI 192.168.202.119@tcp123 [14541.215182] LNet: Removed LNI 192.168.202.119@tcp124 [14542.241405] LNet: Removed LNI 192.168.202.119@tcp125 [14543.266420] LNet: Removed LNI 192.168.202.119@tcp126 [14543.271545] LNet: Removed LNI 192.168.202.119@tcp127 [14544.287136] LNet: Removed LNI 192.168.202.119@tcp128 [14545.311223] LNet: Removed LNI 192.168.202.119@tcp129 [14545.317895] LNet: Removed LNI 192.168.202.119@tcp130 [14545.320847] LNet: Removed LNI 192.168.202.119@tcp131 [14545.328951] LNet: Removed LNI 192.168.202.119@tcp132 [14545.331879] LNet: Removed LNI 192.168.202.119@tcp133 [14545.332985] LNet: Removed LNI 192.168.202.119@tcp134 [14545.349714] LNet: Removed LNI 192.168.202.119@tcp135 [14546.181574] Key type .llcrypt unregistered [14546.183253] Key type ._llcrypt unregistered [14573.342402] Key type ._llcrypt registered [14573.344862] Key type .llcrypt registered [14573.439927] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [14590.363502] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [14590.977559] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14591.103833] alg: No test for adler32 (adler32-zlib) [14592.188404] Lustre: Lustre: Build Version: 2.17.50_30_gbb2e37c [14592.459563] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [14594.175308] Key type lgssc registered [14595.448477] Lustre: Echo OBD driver; http://www.lustre.org/ [14635.411821] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 05:59:31 (1768561171) [14654.117969] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [14657.062946] Key type lgssc unregistered [14657.400689] LNet: 442838:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14657.414101] LNetError: 442838:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [14657.455597] LNet: Removed LNI 192.168.202.119@tcp [14658.547661] Key type .llcrypt unregistered [14658.550563] Key type ._llcrypt unregistered [14674.138986] Key type ._llcrypt registered [14674.140717] Key type .llcrypt registered [14674.238109] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [14675.402605] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14675.560096] alg: No test for adler32 (adler32-zlib) [14676.641265] Lustre: Lustre: Build Version: 2.17.50_30_gbb2e37c [14676.862379] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [14678.561699] Key type lgssc registered [14679.461923] Lustre: Echo OBD driver; http://www.lustre.org/ [14687.987729] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14688.020611] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14689.266600] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14689.323413] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14689.421375] Lustre: lustre-MDT0000: new disk, initializing [14689.506204] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14689.523442] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14693.631594] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [14704.560174] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [14704.687145] Lustre: 444031:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14704.767518] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14704.772510] Lustre: Skipped 1 previous similar message [14704.877539] Lustre: lustre-MDT0001: new disk, initializing [14705.007642] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14705.055375] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14705.073269] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14708.962307] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [14716.876629] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [14720.453632] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [14728.660936] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [14728.894497] Lustre: lustre-OST0000: new disk, initializing [14728.897988] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14728.977115] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14735.183368] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [14738.033388] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14738.043130] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14738.154080] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14742.449221] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [14746.302395] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing lnet_if_list [14752.032340] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [14752.082665] LNet: Added LNI 192.168.202.119@tcp10 [8/256/0/180] [14758.681899] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [14758.721498] LNet: Added LNI 192.168.202.119@tcp11 [8/256/0/180] [14763.326863] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [14763.366385] LNet: Added LNI 192.168.202.119@tcp12 [8/256/0/180] [14768.388865] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [14768.421358] LNet: Added LNI 192.168.202.119@tcp13 [8/256/0/180] [14774.410792] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [14774.445138] LNet: Added LNI 192.168.202.119@tcp14 [8/256/0/180] [14779.759875] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [14779.802453] LNet: Added LNI 192.168.202.119@tcp15 [8/256/0/180] [14784.962355] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [14784.997860] LNet: Added LNI 192.168.202.119@tcp16 [8/256/0/180] [14790.805484] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [14790.837930] LNet: Added LNI 192.168.202.119@tcp17 [8/256/0/180] [14796.580669] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [14796.626779] LNet: Added LNI 192.168.202.119@tcp18 [8/256/0/180] [14802.583623] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [14802.615358] LNet: Added LNI 192.168.202.119@tcp19 [8/256/0/180] [14808.369262] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [14808.405308] LNet: Added LNI 192.168.202.119@tcp20 [8/256/0/180] [14814.180427] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [14814.222596] LNet: Added LNI 192.168.202.119@tcp21 [8/256/0/180] [14820.416629] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [14820.440409] LNet: Added LNI 192.168.202.119@tcp22 [8/256/0/180] [14825.888534] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [14825.908652] LNet: Added LNI 192.168.202.119@tcp23 [8/256/0/180] [14831.926645] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [14831.956700] LNet: Added LNI 192.168.202.119@tcp24 [8/256/0/180] [14837.390715] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [14837.410638] LNet: Added LNI 192.168.202.119@tcp25 [8/256/0/180] [14843.552639] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [14843.607212] LNet: Added LNI 192.168.202.119@tcp26 [8/256/0/180] [14849.974678] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [14850.014463] LNet: Added LNI 192.168.202.119@tcp27 [8/256/0/180] [14850.040915] LNetError: 443289:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.119 [14850.064109] LNetError: Protocol error connecting to 192.168.202.119@tcp26 at host 192.168.202.119:988: is it running a compatible version of Lustre? [14856.999457] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [14857.035865] LNet: Added LNI 192.168.202.119@tcp28 [8/256/0/180] [14857.048397] LNetError: 443290:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.119 [14857.073525] LNetError: Protocol error connecting to 192.168.202.119@tcp26 at host 192.168.202.119:988: is it running a compatible version of Lustre? [14863.951666] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [14863.986716] LNet: Added LNI 192.168.202.119@tcp29 [8/256/0/180] [14863.993521] LNetError: 443292:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.119 [14864.021893] LNetError: Protocol error connecting to 192.168.202.119@tcp28 at host 192.168.202.119:988: is it running a compatible version of Lustre? [14864.287088] LNet: 1 local NIs in recovery (showing 1): 192.168.202.119@tcp28 [14867.361885] LNetError: 443291:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.119 [14867.374797] LNetError: 443291:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 1 previous similar message [14867.384596] LNetError: Protocol error connecting to 192.168.202.119@tcp28 at host 192.168.202.119:988: is it running a compatible version of Lustre? [14867.397695] LNetError: Skipped 1 previous similar message [14870.358617] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [14870.411500] LNet: Added LNI 192.168.202.119@tcp30 [8/256/0/180] [14871.456546] LNetError: 443289:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.119 [14871.465715] LNetError: 443289:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 2 previous similar messages [14871.473991] LNetError: Protocol error connecting to 192.168.202.119@tcp28 at host 192.168.202.119:988: is it running a compatible version of Lustre? [14871.482245] LNetError: Skipped 3 previous similar messages [14877.512514] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [14877.541705] LNet: Added LNI 192.168.202.119@tcp31 [8/256/0/180] [14879.655479] LNetError: 443291:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.119 [14879.670265] LNetError: 443291:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 3 previous similar messages [14879.679710] LNetError: Protocol error connecting to 192.168.202.119@tcp28 at host 192.168.202.119:988: is it running a compatible version of Lustre? [14879.695871] LNetError: Skipped 2 previous similar messages [14884.389915] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [14884.409441] LNet: Added LNI 192.168.202.119@tcp32 [8/256/0/180] [14891.294609] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [14891.320518] LNet: Added LNI 192.168.202.119@tcp33 [8/256/0/180] [14899.019359] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [14899.039846] LNet: Added LNI 192.168.202.119@tcp34 [8/256/0/180] [14899.052785] LNetError: 443292:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.119 [14899.061250] LNetError: 443292:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 8 previous similar messages [14899.080240] LNetError: Protocol error connecting to 192.168.202.119@tcp32 at host 192.168.202.119:988: is it running a compatible version of Lustre? [14899.091591] LNetError: Skipped 8 previous similar messages [14906.178974] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [14906.205637] LNet: Added LNI 192.168.202.119@tcp35 [8/256/0/180] [14913.402637] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [14913.446660] LNet: Added LNI 192.168.202.119@tcp36 [8/256/0/180] [14920.303798] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [14920.339663] LNet: Added LNI 192.168.202.119@tcp37 [8/256/0/180] [14923.684520] LNet: 6 local NIs in recovery (showing 6): 192.168.202.119@tcp35, 192.168.202.119@tcp32, 192.168.202.119@tcp33, 192.168.202.119@tcp30, 192.168.202.119@tcp31, 192.168.202.119@tcp28 [14927.994490] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [14928.040704] LNet: Added LNI 192.168.202.119@tcp38 [8/256/0/180] [14933.986758] LNetError: 443289:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.119 [14934.010548] LNetError: 443289:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 34 previous similar messages [14934.018370] LNetError: Protocol error connecting to 192.168.202.119@tcp36 at host 192.168.202.119:988: is it running a compatible version of Lustre? [14934.028224] LNetError: Skipped 34 previous similar messages [14935.295355] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [14935.327195] LNet: Added LNI 192.168.202.119@tcp39 [8/256/0/180] [14944.361797] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [14944.398795] LNet: Added LNI 192.168.202.119@tcp40 [8/256/0/180] [14953.114584] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [14953.151277] LNet: Added LNI 192.168.202.119@tcp41 [8/256/0/180] [14961.685663] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [14961.705202] LNet: Added LNI 192.168.202.119@tcp42 [8/256/0/180] [14970.138618] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [14970.170915] LNet: Added LNI 192.168.202.119@tcp43 [8/256/0/180] [14981.372710] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [14981.416351] LNet: Added LNI 192.168.202.119@tcp44 [8/256/0/180] [14990.661668] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [14990.705899] LNet: Added LNI 192.168.202.119@tcp45 [8/256/0/180] [14998.501908] LNetError: 443292:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.119 [14998.531256] LNetError: 443292:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 50 previous similar messages [14998.543666] LNetError: Protocol error connecting to 192.168.202.119@tcp38 at host 192.168.202.119:988: is it running a compatible version of Lustre? [14998.549597] LNetError: Skipped 50 previous similar messages [14999.920191] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [14999.942122] LNet: Added LNI 192.168.202.119@tcp46 [8/256/0/180] [15008.457977] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [15008.488975] LNet: Added LNI 192.168.202.119@tcp47 [8/256/0/180] [15016.805440] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [15016.846231] LNet: Added LNI 192.168.202.119@tcp48 [8/256/0/180] [15026.264698] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [15026.311049] LNet: Added LNI 192.168.202.119@tcp49 [8/256/0/180] [15035.408478] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [15035.444587] LNet: Added LNI 192.168.202.119@tcp50 [8/256/0/180] [15044.581518] LNet: 14 local NIs in recovery (showing 14): 192.168.202.119@tcp48, 192.168.202.119@tcp35, 192.168.202.119@tcp46, 192.168.202.119@tcp47, 192.168.202.119@tcp45, 192.168.202.119@tcp32, 192.168.202.119@tcp33, 192.168.202.119@tcp31, 192.168.202.119@tcp40, 192.168.202.119@tcp43, 192.168.202.119@tcp30, 192.168.202.119@tcp38, 192.168.202.119@tcp28, 192.168.202.119@tcp36 [15055.842116] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15055.847541] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15055.854856] LustreError: Skipped 1 previous similar message [15055.857719] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15055.865501] Lustre: Skipped 1 previous similar message [15055.879643] Lustre: Skipped 1 previous similar message [15060.977417] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15060.992108] Lustre: Skipped 1 previous similar message [15061.766720] LustreError: 468330:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [15061.996918] Lustre: server umount lustre-OST0000 complete [15068.642590] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15068.659303] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15068.679908] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15071.204023] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15071.212787] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15073.025617] LustreError: 468531:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [15073.028309] LustreError: 468531:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [15073.289920] Lustre: server umount lustre-MDT0000 complete [15077.405334] LustreError: 444020:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768561614 with bad export cookie 8725029736382065559 [15077.407690] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15077.410974] LustreError: 444020:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [15077.599933] LustreError: 468734:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [15077.605927] LustreError: 468734:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [15077.792601] Lustre: server umount lustre-MDT0001 complete [15088.709077] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [15091.502939] Key type lgssc unregistered [15091.901515] LNet: 469274:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15091.907649] LNetError: 469274:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [15091.915794] LNet: Removed LNI 192.168.202.119@tcp [15092.979219] LNet: Removed LNI 192.168.202.119@tcp10 [15092.985675] LNetError: 469274:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [15092.998865] LNetError: 469274:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [15093.021163] LNet: Removed LNI 192.168.202.119@tcp11 [15094.047207] LNet: Removed LNI 192.168.202.119@tcp12 [15094.049230] LNetError: 469274:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [15094.060336] LNetError: 469274:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [15094.073939] LNet: Removed LNI 192.168.202.119@tcp13 [15095.135150] LNet: Removed LNI 192.168.202.119@tcp14 [15096.159748] LNet: Removed LNI 192.168.202.119@tcp15 [15096.174254] LNetError: 469274:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [15096.194754] LNetError: 469274:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [15096.213019] LNet: Removed LNI 192.168.202.119@tcp16 [15097.249600] LNet: Removed LNI 192.168.202.119@tcp17 [15098.271174] LNet: Removed LNI 192.168.202.119@tcp18 [15099.295962] LNet: Removed LNI 192.168.202.119@tcp19 [15100.319779] LNet: Removed LNI 192.168.202.119@tcp20 [15100.328500] LNetError: 469274:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [15100.337928] LNetError: 469274:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [15100.347772] LNet: Removed LNI 192.168.202.119@tcp21 [15101.408820] LNet: Removed LNI 192.168.202.119@tcp22 [15102.431196] LNet: Removed LNI 192.168.202.119@tcp23 [15103.456436] LNet: Removed LNI 192.168.202.119@tcp24 [15104.484979] LNet: Removed LNI 192.168.202.119@tcp25 [15105.503319] LNet: Removed LNI 192.168.202.119@tcp26 [15106.528531] LNet: Removed LNI 192.168.202.119@tcp27 [15106.530269] LNet: Removed LNI 192.168.202.119@tcp28 [15107.617144] LNet: Removed LNI 192.168.202.119@tcp29 [15107.620646] LNet: Removed LNI 192.168.202.119@tcp30 [15107.629621] LNet: Removed LNI 192.168.202.119@tcp31 [15107.637089] LNet: Removed LNI 192.168.202.119@tcp32 [15107.642312] LNet: Removed LNI 192.168.202.119@tcp33 [15108.703787] LNet: Removed LNI 192.168.202.119@tcp34 [15108.712779] LNetError: 469274:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [15108.717552] LNetError: 469274:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 13 previous similar messages [15108.727301] LNet: Removed LNI 192.168.202.119@tcp35 [15108.730520] LNet: Removed LNI 192.168.202.119@tcp36 [15109.791255] LNet: Removed LNI 192.168.202.119@tcp37 [15109.792949] LNet: Removed LNI 192.168.202.119@tcp38 [15110.815459] LNet: Removed LNI 192.168.202.119@tcp39 [15110.819156] LNet: Removed LNI 192.168.202.119@tcp40 [15111.839221] LNet: Removed LNI 192.168.202.119@tcp41 [15112.865504] LNet: Removed LNI 192.168.202.119@tcp42 [15112.870746] LNet: Removed LNI 192.168.202.119@tcp43 [15113.887467] LNet: Removed LNI 192.168.202.119@tcp44 [15113.893236] LNet: Removed LNI 192.168.202.119@tcp45 [15113.908549] LNet: Removed LNI 192.168.202.119@tcp46 [15114.976614] LNet: Removed LNI 192.168.202.119@tcp47 [15114.979869] LNet: Removed LNI 192.168.202.119@tcp48 [15114.985563] LNet: Removed LNI 192.168.202.119@tcp49 [15115.021225] LNet: Removed LNI 192.168.202.119@tcp50 [15115.719307] Key type .llcrypt unregistered [15115.729730] Key type ._llcrypt unregistered [15138.289401] Key type ._llcrypt registered [15138.292352] Key type .llcrypt registered [15138.411028] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [15140.056167] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15140.084515] alg: No test for adler32 (adler32-zlib) [15141.246769] Lustre: Lustre: Build Version: 2.17.50_30_gbb2e37c [15141.543988] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [15143.335421] Key type lgssc registered [15144.296126] Lustre: Echo OBD driver; http://www.lustre.org/ [15154.120782] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15154.160679] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15155.802039] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15161.075297] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [15165.927507] LustreError: 470523:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15165.950610] LustreError: 470523:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [15166.946708] LustreError: 470523:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15171.480738] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [15171.867245] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15176.643704] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [15183.817108] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [15189.114627] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [15196.882262] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [15197.402395] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15204.348338] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [15211.712412] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [15225.930568] loop0: detected capacity change from 0 to 204800000 [15226.072027] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15226.082850] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15226.105723] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15233.137356] loop0: detected capacity change from 0 to 204800000 [15233.410226] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15233.435379] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15233.445575] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15240.186115] loop0: detected capacity change from 0 to 204800000 [15246.461682] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [15246.659155] Lustre: fs15246-OST0000: new disk, initializing [15246.668307] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [15246.714920] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15254.445170] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [15265.397926] loop1: detected capacity change from 0 to 204800000 [15272.433623] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [15272.623622] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [15272.691793] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [15272.759219] Lustre: fs15246-MDT0000: new disk, initializing [15272.932350] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15272.961120] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15273.087329] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [15274.149482] Lustre: 474885:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@00000000b60f723c x1854471330119424/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1768561827 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [15278.451530] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [15283.238496] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [15283.254093] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [15283.297477] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [15302.645045] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15302.690097] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [15307.753410] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [15307.879891] LustreError: 476009:0:(obd_class.h:479:obd_check_dev()) Device 30 not setup [15307.981495] Lustre: server umount fs15246-MDT0000 complete [15312.876137] LustreError: 471633:0:(ldlm_lib.c:1179:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15318.002262] LustreError: 470517:0:(ldlm_lib.c:1179:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15318.830193] LustreError: 476510:0:(obd_class.h:479:obd_check_dev()) Device 32 not setup [15318.832874] LustreError: 476510:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [15318.889963] Lustre: server umount fs15246-OST0000 complete [15330.792502] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15330.804237] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15330.823549] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15333.346724] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15333.347122] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15336.001169] LustreError: 477013:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [15336.011141] LustreError: 477013:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [15336.202306] Lustre: server umount lustre-OST0000 complete [15341.033197] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15341.044487] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15341.083680] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15341.102985] Lustre: Skipped 1 previous similar message [15346.474069] LustreError: 477215:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [15346.496058] LustreError: 477215:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [15346.687144] Lustre: server umount lustre-MDT0000 complete [15348.704806] LustreError: 470517:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15348.721601] LustreError: 470517:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [15351.145966] LustreError: 470503:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768561888 with bad export cookie 1165330352474257216 [15351.153063] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15351.159035] LustreError: 470503:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [15351.340523] LustreError: 477417:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [15351.356280] LustreError: 477417:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [15351.542868] Lustre: server umount lustre-MDT0001 complete [15364.986185] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [15368.886596] Key type lgssc unregistered [15369.211931] LNet: 477949:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15369.223613] LNetError: 477949:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [15370.281796] LNet: Removed LNI 192.168.202.119@tcp [15371.443522] Key type .llcrypt unregistered [15371.447424] Key type ._llcrypt unregistered [15394.343758] Key type ._llcrypt registered [15394.345279] Key type .llcrypt registered [15394.452449] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [15395.544477] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15395.568278] alg: No test for adler32 (adler32-zlib) [15396.519256] Lustre: Lustre: Build Version: 2.17.50_30_gbb2e37c [15396.739882] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [15398.392164] Key type lgssc registered [15399.811719] Lustre: Echo OBD driver; http://www.lustre.org/ [15450.999229] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [15461.382090] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [15507.264603] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15507.329057] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15509.034233] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15509.149598] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15509.239967] Lustre: lustre-MDT0000: new disk, initializing [15509.326375] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15509.357717] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15515.148782] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [15528.611665] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [15528.807128] Lustre: 482370:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15528.901832] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15528.906544] Lustre: Skipped 1 previous similar message [15529.000346] Lustre: lustre-MDT0001: new disk, initializing [15529.091140] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15529.139382] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15529.144848] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15533.634419] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [15541.438844] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [15545.648137] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [15555.194802] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [15555.447633] Lustre: lustre-OST0000: new disk, initializing [15555.450909] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15555.543528] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15565.444658] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15565.459362] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15565.577042] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15566.212540] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [15574.541631] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [15579.411939] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [15579.732710] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [15585.369931] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [15585.589419] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [15591.412816] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15591.426399] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15591.456888] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15596.519972] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15596.525449] Lustre: Skipped 2 previous similar messages [15601.648149] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15602.143258] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15602.212584] LustreError: 485304:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [15602.423303] Lustre: server umount lustre-OST0000 complete [15611.365677] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15611.371556] LustreError: Skipped 1 previous similar message [15611.376315] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15611.386196] Lustre: Skipped 1 previous similar message [15611.392189] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15611.394562] Lustre: Skipped 1 previous similar message [15615.662396] LustreError: 485508:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [15615.679872] LustreError: 485508:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [15615.949985] Lustre: server umount lustre-MDT0000 complete [15621.541666] LustreError: 482363:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768562158 with bad export cookie 3386109148867800633 [15621.552718] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15621.554073] LustreError: 482363:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [15621.750222] LustreError: 485710:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [15621.753098] LustreError: 485710:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [15622.063429] Lustre: server umount lustre-MDT0001 complete [15655.711948] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [15670.099406] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [15713.752668] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 15485 sec ========= 06:17:28 (1768562248) [15715.983081] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 06:17:31 (1768562251) === [15720.700872] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 06:17:36 (1768562256) === [15770.498153] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [15774.279151] Key type lgssc unregistered [15774.693973] LNet: 490562:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15774.700312] LNetError: 490562:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [15774.720354] LNet: Removed LNI 192.168.202.119@tcp [15775.811767] Key type .llcrypt unregistered [15775.813375] Key type ._llcrypt unregistered