[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 1000238127 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2439 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22D5 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 002295 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2349 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23D9 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2411 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22d5-0xbffe2348] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22d4] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2349-0xbffe23d8] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23d9-0xbffe2410] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2411-0xbffe2438] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001008] APIC: Switch to symmetric I/O mode setup [ 0.002183] x2apic enabled [ 0.003005] Switched APIC routing to physical x2apic. [ 0.004010] kvm-guest: setup PV IPIs [ 0.006597] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007019] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008012] pid_max: default: 32768 minimum: 301 [ 0.009175] LSM: Security Framework initializing [ 0.010050] Yama: becoming mindful. [ 0.011055] SELinux: Initializing. [ 0.012075] *** VALIDATE selinux *** [ 0.022662] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.030461] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.031147] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032190] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.033106] *** VALIDATE tmpfs *** [ 0.034420] *** VALIDATE proc *** [ 0.036297] *** VALIDATE cgroup *** [ 0.037008] *** VALIDATE cgroup2 *** [ 0.039104] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.041140] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.042007] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.043027] Spectre V2 : User space: Vulnerable [ 0.044006] Speculative Store Bypass: Vulnerable [ 0.047687] debug: unmapping init [mem 0xffffffffbc259000-0xffffffffbc260fff] [ 0.049272] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.050769] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.051021] ... version: 2 [ 0.052009] ... bit width: 48 [ 0.053010] ... generic registers: 4 [ 0.054011] ... value mask: 0000ffffffffffff [ 0.055009] ... max period: 00007fffffffffff [ 0.056013] ... fixed-purpose events: 3 [ 0.057012] ... event mask: 000000070000000f [ 0.058286] rcu: Hierarchical SRCU implementation. [ 0.060529] smp: Bringing up secondary CPUs ... [ 0.061704] x86: Booting SMP configuration: [ 0.062024] .... node #0, CPUs: #1 #2 #3 [ 0.156060] smp: Brought up 1 node, 4 CPUs [ 0.158013] smpboot: Max logical packages: 1 [ 0.159009] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.237024] node 0 deferred pages initialised in 76ms [ 0.247705] devtmpfs: initialized [ 0.248000] x86/mm: Memory block size: 128MB [ 0.261866] gcov: version magic: 0x41383552 [ 0.276000] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.282618] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.288610] pinctrl core: initialized pinctrl subsystem [ 0.293076] [ 0.295395] ************************************************************* [ 0.301014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.305013] ** ** [ 0.308018] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.311013] ** ** [ 0.315100] ** This means that this kernel is built to expose internal ** [ 0.320014] ** IOMMU data structures, which may compromise security on ** [ 0.322013] ** your system. ** [ 0.326013] ** ** [ 0.330014] ** If you see this message and you are not debugging the ** [ 0.333012] ** kernel, report this immediately to your vendor! ** [ 0.336013] ** ** [ 0.342014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.347024] ************************************************************* [ 0.350835] NET: Registered protocol family 16 [ 0.353511] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.357060] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.361064] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.365020] cpuidle: using governor menu [ 0.366863] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.370475] PCI: Using configuration type 1 for base access [ 0.373138] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.387446] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.389088] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.392041] cryptd: max_cpu_qlen set to 1000 [ 0.394237] ACPI: Added _OSI(Module Device) [ 0.395000] ACPI: Added _OSI(Processor Device) [ 0.397013] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.400017] ACPI: Added _OSI(Processor Aggregator Device) [ 0.406994] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.418439] ACPI: Interpreter enabled [ 0.422073] ACPI: PM: (supports S0 S3 S4 S5) [ 0.425019] ACPI: Using IOAPIC for interrupt routing [ 0.429057] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.433000] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.454592] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.460042] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.465030] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.471123] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.541000] acpiphp: Slot [2] registered [ 0.543561] acpiphp: Slot [5] registered [ 0.546503] acpiphp: Slot [6] registered [ 0.549396] acpiphp: Slot [7] registered [ 0.552488] acpiphp: Slot [8] registered [ 0.554141] acpiphp: Slot [9] registered [ 0.556291] acpiphp: Slot [10] registered [ 0.558328] acpiphp: Slot [3] registered [ 0.561121] acpiphp: Slot [4] registered [ 0.563135] acpiphp: Slot [11] registered [ 0.565136] acpiphp: Slot [12] registered [ 0.567106] acpiphp: Slot [13] registered [ 0.570122] acpiphp: Slot [14] registered [ 0.573129] acpiphp: Slot [15] registered [ 0.575000] acpiphp: Slot [16] registered [ 0.575000] acpiphp: Slot [17] registered [ 0.575000] acpiphp: Slot [18] registered [ 0.575000] acpiphp: Slot [19] registered [ 0.575000] acpiphp: Slot [20] registered [ 0.575000] acpiphp: Slot [21] registered [ 0.575000] acpiphp: Slot [22] registered [ 0.575000] acpiphp: Slot [23] registered [ 0.575000] acpiphp: Slot [24] registered [ 0.575000] acpiphp: Slot [25] registered [ 0.575000] acpiphp: Slot [26] registered [ 0.575000] acpiphp: Slot [27] registered [ 0.575000] acpiphp: Slot [28] registered [ 0.575000] acpiphp: Slot [29] registered [ 0.575000] acpiphp: Slot [30] registered [ 0.575000] acpiphp: Slot [31] registered [ 0.575000] PCI host bridge to bus 0000:00 [ 0.575000] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.575000] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.575000] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.575000] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.575000] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.575000] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.575000] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.575000] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.575000] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.575000] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.575000] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.575000] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.575000] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.575000] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.575000] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.575000] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.575000] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.584000] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.594025] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.615018] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.622018] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.631363] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.649022] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.661018] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.681016] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.689802] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.698013] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.706028] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.728016] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.823037] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.836015] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.868015] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.883019] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.891740] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.901015] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.914014] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.963016] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.978388] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.988015] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.997017] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 1.008000] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 1.021611] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 1.069025] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 1.104016] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 1.129015] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 1.141719] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 1.144456] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 1.147356] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 1.150408] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 1.152166] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 1.156136] iommu: Default domain type: Passthrough [ 1.160419] SCSI subsystem initialized [ 1.161214] ACPI: bus type USB registered [ 1.162087] usbcore: registered new interface driver usbfs [ 1.164110] usbcore: registered new interface driver hub [ 1.166062] usbcore: registered new device driver usb [ 1.168237] pps_core: LinuxPPS API ver. 1 registered [ 1.170011] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 1.173212] PTP clock support registered [ 1.176331] EDAC MC: Ver: 3.0.0 [ 1.177124] PCI: Using ACPI for IRQ routing [ 1.180071] NetLabel: Initializing [ 1.181008] NetLabel: domain hash size = 128 [ 1.183010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 1.186081] NetLabel: unlabeled traffic allowed by default [ 1.189358] vgaarb: loaded [ 1.190495] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 1.193010] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 1.203466] clocksource: Switched to clocksource kvm-clock [ 1.395748] VFS: Disk quotas dquot_6.6.0 [ 1.397595] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.402539] *** VALIDATE ramfs *** [ 1.405447] *** VALIDATE hugetlbfs *** [ 1.408682] pnp: PnP ACPI init [ 1.413723] pnp: PnP ACPI: found 6 devices [ 1.457521] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.461890] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.465057] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.467687] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.471287] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.474522] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.478734] NET: Registered protocol family 2 [ 1.481710] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.531139] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.537333] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.591972] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.601307] TCP: Hash tables configured (established 65536 bind 65536) [ 1.746894] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.760075] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.766747] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.773767] NET: Registered protocol family 1 [ 1.777835] RPC: Registered named UNIX socket transport module. [ 1.780868] RPC: Registered udp transport module. [ 1.782916] RPC: Registered tcp transport module. [ 1.785376] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.788099] NET: Registered protocol family 44 [ 1.789338] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.791756] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.793896] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.796355] PCI: CLS 0 bytes, default 64 [ 1.797885] Unpacking initramfs... [ 3.729605] debug: unmapping init [mem 0xffff9e6f3cc54000-0xffff9e6f3ffbffff] [ 3.735455] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 3.738792] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 3.743680] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 6.139472] Initialise system trusted keyrings [ 6.142557] Key type blacklist registered [ 6.145372] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 6.159083] zbud: loaded [ 6.162653] *** VALIDATE nfs *** [ 6.164454] *** VALIDATE nfs4 *** [ 6.166698] pstore: using deflate compression [ 6.171679] Platform Keyring initialized [ 6.320982] NET: Registered protocol family 38 [ 6.323211] Key type asymmetric registered [ 6.326166] Asymmetric key parser 'x509' registered [ 6.328295] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 6.331996] io scheduler mq-deadline registered [ 6.333486] io scheduler kyber registered [ 6.334772] io scheduler bfq registered [ 6.337214] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 6.340266] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 6.342482] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 6.346208] ACPI: Power Button [PWRF] [ 6.352052] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 6.360685] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 6.470787] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 6.482550] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 6.503838] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 6.568881] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 6.599501] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 6.603820] Non-volatile memory driver v1.3 [ 6.605737] Linux agpgart interface v0.103 [ 6.640452] virtio_blk virtio1: [vda] 134032 512-byte logical blocks (68.6 MB/65.4 MiB) [ 6.644137] vda: detected capacity change from 0 to 68624384 [ 6.705384] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 6.710160] vdb: detected capacity change from 0 to 1073741824 [ 6.740770] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 6.745421] vdc: detected capacity change from 0 to 2621440000 [ 6.770803] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 6.775358] vdd: detected capacity change from 0 to 2621440000 [ 6.800478] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 6.804449] vde: detected capacity change from 0 to 4294967296 [ 6.851545] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 6.859074] vdf: detected capacity change from 0 to 4294967296 [ 6.873371] libphy: Fixed MDIO Bus: probed [ 6.882548] usbcore: registered new interface driver usbserial_generic [ 6.885370] usbserial: USB Serial support registered for generic [ 6.888105] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 6.893464] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 6.896067] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 6.899561] mousedev: PS/2 mouse device common for all mice [ 6.904769] rtc_cmos 00:05: RTC can wake from S4 [ 6.912782] rtc_cmos 00:05: registered as rtc0 [ 6.916590] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 6.924280] intel_pstate: CPU model not supported [ 6.927296] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 6.933037] hid: raw HID events driver (C) Jiri Kosina [ 6.935494] usbcore: registered new interface driver usbhid [ 6.937778] usbhid: USB HID core driver [ 6.939524] drop_monitor: Initializing network drop monitor service [ 6.942453] Initializing XFRM netlink socket [ 6.945265] NET: Registered protocol family 10 [ 6.946846] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 6.949794] Segment Routing with IPv6 [ 6.958487] NET: Registered protocol family 17 [ 6.960234] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 6.961773] mpls_gso: MPLS GSO support [ 6.984580] RAS: Correctable Errors collector initialized. [ 6.988396] AVX version of gcm_enc/dec engaged. [ 6.990835] AES CTR mode by8 optimization enabled [ 7.120269] sched_clock: Marking stable (7120242986, 0)->(8546719612, -1426476626) [ 7.140315] registered taskstats version 1 [ 7.147146] Loading compiled-in X.509 certificates [ 7.162415] zswap: loaded using pool lzo/zbud [ 7.311724] Key type big_key registered [ 7.351287] Key type encrypted registered [ 7.354668] ima: No TPM chip found, activating TPM-bypass! [ 7.357963] ima: Allocated hash algorithm: sha1 [ 7.360208] ima: No architecture policies found [ 7.362628] evm: Initialising EVM extended attributes: [ 7.365420] evm: security.selinux [ 7.367088] evm: security.ima [ 7.368610] evm: security.capability [ 7.370427] evm: HMAC attrs: 0x1 [ 7.373672] rtc_cmos 00:05: setting system clock to 2026-01-02 00:49:34 UTC (1767314974) [ 7.380504] debug: unmapping init [mem 0xffffffffbd203000-0xffffffffbd3fffff] [ 7.384592] debug: unmapping init [mem 0xffffffffbbf82000-0xffffffffbc258fff] [ 7.395261] Write protecting the kernel read-only data: 28672k [ 7.399480] debug: unmapping init [mem 0xffffffffba603000-0xffffffffba7fffff] [ 7.402398] debug: unmapping init [mem 0xffffffffbaf14000-0xffffffffbaffffff] [ 7.458288] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 7.480444] systemd[1]: Detected virtualization kvm. [ 7.482620] systemd[1]: Detected architecture x86-64. [ 7.484608] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 7.512404] systemd[1]: No hostname configured. [ 7.514040] systemd[1]: Set hostname to . [ 7.516154] random: systemd: uninitialized urandom read (16 bytes read) [ 7.518406] systemd[1]: Initializing machine ID from random generator. [ 7.628680] random: ln: uninitialized urandom read (6 bytes read) [ 8.046638] random: systemd: uninitialized urandom read (16 bytes read) [ 8.049343] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 8.053821] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 8.061338] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Swap. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Timers. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... Starting Apply Kernel Variables... Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 11.350722] device-mapper: uevent: version 1.0.3 [ 11.353638] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ 13.599233] hrtimer: interrupt took 2282061 ns [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ 15.104357] random: fast init done [ 15.318788] virtio_net virtio0 ens2: renamed from eth0 [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 17.934674] scsi host0: ata_piix [ 18.177591] scsi host1: ata_piix [ 18.195853] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 18.201434] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 24.113297] random: crng init done [ 24.115848] random: 7 urandom warning(s) missed due to ratelimiting [ 26.721484] dracut-initqueue[585]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 30.168406] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 33.600473] printk: systemd: 26 output lines suppressed due to ratelimiting [ 34.973880] SELinux: Disabled at runtime. [ 35.168866] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 35.220988] systemd[1]: Detected virtualization kvm. [ 35.226754] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 37.424280] systemd[1]: initrd-switch-root.service: Succeeded. [ 37.438405] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 37.463941] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 37.474866] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 37.488617] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 37.513711] systemd[1]: Starting Journal Service... Starting Journal Service... [ 37.529622] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting POSIX Message Queue File System... [ OK ] Created slice system-getty.slice. [ OK ] Reached target Paths. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Starting Remount Root and Kernel File Systems... Mounting Huge Pages File System... [ OK ] Reached target Local Encrypted Volumes. [ 38.266207] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting Kernel Debug File System... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. Starting Apply Kernel Variables... [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 39.773084] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 41.035284] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 41.196981] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 41.968933] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 42.131909] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (8s / no limit) [** ] A start job is running for Configur…-only root support (9s / no limit) [*** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…only root support (10s / no limit) [ *** ] A start job is running for Configur…only root support (10s / no limit) [ ***] A start job is running for Configur…only root support (11s / no limit)[ 48.322774] Key type dns_resolver registered [ **] A start job is running for Configur…only root support (11s / no limit) [ *] A start job is running for Configur…only root support (12s / no limit)[ 49.386237] NFS: Registering the id_resolver key type [ 49.388122] Key type id_resolver registered [ 49.390129] Key type id_legacy registered [ **] A start job is running for Configur…only root support (12s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. Starting Restore /run/initramfs on shutdown... Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg352-server login: [ 117.352598] libcfs: loading out-of-tree module taints kernel. [ 117.438569] Key type ._llcrypt registered [ 117.442160] Key type .llcrypt registered [ 117.552044] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_hostid [ 194.022787] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 195.439498] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 195.481505] alg: No test for adler32 (adler32-zlib) [ 196.894855] Lustre: Lustre: Build Version: 2.17.0_RC4_1_g3742a8b [ 197.893563] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [ 199.736371] Key type lgssc registered [ 201.534957] Lustre: Echo OBD driver; http://www.lustre.org/ [ 229.494862] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 231.063648] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 244.936425] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 254.769889] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 266.334930] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 351.312339] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 372.820682] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 372.877505] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 372.892272] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 374.338425] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 374.373914] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 374.484761] Lustre: lustre-MDT0000: new disk, initializing [ 374.613439] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 374.635353] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 378.717863] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 386.404852] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 402.981848] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 403.032859] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 403.117335] Lustre: 14442:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 403.157961] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 403.162350] Lustre: Skipped 1 previous similar message [ 403.271247] Lustre: lustre-MDT0001: new disk, initializing [ 403.320282] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 403.348385] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 403.366749] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 407.621797] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 415.177909] Lustre: 16503:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 433.642491] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 433.791044] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 434.201336] Lustre: lustre-OST0000: new disk, initializing [ 434.212726] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 434.358962] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 440.365264] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 440.380916] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 440.529981] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 443.143795] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 465.002240] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 465.075775] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 465.153665] Lustre: lustre-OST0001: new disk, initializing [ 465.157719] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 465.227700] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 471.690427] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 473.622930] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 473.637498] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 473.698109] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 484.430309] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 490.788641] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 502.714553] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing check_logdir /tmp/testlogs/ [ 508.540729] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing yml_node [ 517.525879] Lustre: DEBUG MARKER: Client: 2.17.0.RC4 [ 520.009552] Lustre: DEBUG MARKER: MDS: 2.17.0.RC4 [ 522.758921] Lustre: DEBUG MARKER: OSS: 2.17.0.RC4 [ 524.587748] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Jan 1 19:58:10 EST 2026 [ 548.215989] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 550.011513] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 567.266484] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 567.279266] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 567.302195] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 570.849140] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 570.850075] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 570.860921] Lustre: Skipped 2 previous similar messages [ 570.876619] Lustre: Skipped 2 previous similar messages [ 572.499456] LustreError: 21893:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 572.619479] Lustre: server umount lustre-MDT0000 complete [ 581.093450] LustreError: 16316:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 581.119624] LustreError: 16316:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 581.180402] LustreError: 14435:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767315548 with bad export cookie 7762012990902912100 [ 581.186397] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 581.191724] LustreError: 14435:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 586.208943] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 586.216353] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 586.217886] LustreError: 14449:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 586.220832] Lustre: Skipped 1 previous similar message [ 586.258214] Lustre: Skipped 1 previous similar message [ 587.442208] LustreError: 22344:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 587.459198] LustreError: 22344:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 587.639239] Lustre: server umount lustre-MDT0001 complete [ 598.889361] LustreError: 22941:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 598.894928] LustreError: 22941:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 598.921658] Lustre: server umount lustre-OST0000 complete [ 607.978473] LustreError: 23391:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 607.987324] LustreError: 23391:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 608.151404] Lustre: server umount lustre-OST0001 complete [ 622.370994] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_hostid [ 707.154934] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 729.909415] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 742.335887] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 754.443844] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 765.072718] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 782.684758] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 782.785062] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 783.064376] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 783.102248] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 783.190983] Lustre: lustre-MDT0000: new disk, initializing [ 783.277308] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 783.313116] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 788.125065] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 809.534429] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 809.602678] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 809.692921] Lustre: 30620:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 809.750505] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 809.755162] Lustre: Skipped 1 previous similar message [ 809.835643] Lustre: lustre-MDT0001: new disk, initializing [ 809.957651] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 810.002062] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 810.027939] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 813.958729] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 822.682931] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 826.435838] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 840.119321] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 840.244037] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 840.790508] Lustre: lustre-OST0000: new disk, initializing [ 840.800272] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 840.910326] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 842.379420] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 842.387977] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 842.534879] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 847.090672] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 855.402800] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 860.280195] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 860.496736] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 864.830934] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 865.048732] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 868.332467] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 868.335310] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 868.340627] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 868.376137] Lustre: Skipped 1 previous similar message [ 873.040565] LustreError: 34251:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 873.047995] LustreError: 34251:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 873.239683] Lustre: server umount lustre-OST0000 complete [ 881.634195] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 881.638971] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 881.650209] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 881.655260] Lustre: Skipped 1 previous similar message [ 884.887324] LustreError: 34501:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 884.892743] LustreError: 34501:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 885.038535] Lustre: server umount lustre-MDT0000 complete [ 888.809568] LustreError: 30627:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 888.831475] LustreError: 30627:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 890.019154] LustreError: 30612:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767315857 with bad export cookie 7762012990902913647 [ 890.035512] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 890.036536] LustreError: 30612:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 890.318162] Lustre: server umount lustre-MDT0001 complete [ 894.567589] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 20:04:20 (1767315860) [ 898.239614] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 899.773897] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 20:04:25 (1767315865) [ 910.552787] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 910.934359] LustreError: 35662:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 911.076311] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 915.621657] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 916.453657] LustreError: 35661:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 916.475550] LustreError: 35661:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 926.689142] LustreError: 35662:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 926.713328] LustreError: 35662:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 927.762619] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 932.902251] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 938.157291] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 942.328639] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 954.302930] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 961.316939] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 968.609332] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 982.502306] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 982.514633] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 982.535878] Lustre: Skipped 1 previous similar message [ 982.543387] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 982.546141] Lustre: Skipped 2 previous similar messages [ 986.848196] LustreError: 38294:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 986.860409] LustreError: 38294:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 987.087110] Lustre: server umount lustre-OST0000 complete [ 998.740612] Lustre: server umount lustre-MDT0000 complete [ 998.881949] LustreError: 35661:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 998.944748] LustreError: 35661:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1003.807667] LustreError: 35642:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767315970 with bad export cookie 7762012990902914536 [ 1003.824656] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1004.150903] Lustre: server umount lustre-MDT0001 complete [ 1017.888490] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [ 1020.615453] Key type lgssc unregistered [ 1020.854794] LNet: 39483:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1020.861496] LNetError: 39483:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1020.879135] LNet: Removed LNI 192.168.203.152@tcp [ 1021.685397] Key type .llcrypt unregistered [ 1021.687604] Key type ._llcrypt unregistered [ 1100.856755] Key type ._llcrypt registered [ 1100.860313] Key type .llcrypt registered [ 1100.961268] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 1102.003261] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1102.131904] alg: No test for adler32 (adler32-zlib) [ 1103.179472] Lustre: Lustre: Build Version: 2.17.0_RC4_1_g3742a8b [ 1103.384946] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [ 1105.009946] Key type lgssc registered [ 1106.184171] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1116.379086] loop: module loaded [ 1116.486272] loop0: detected capacity change from 0 to 204800000 [ 1116.741373] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 1116.747535] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 1117.080399] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 1117.146794] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1122.945560] loop0: detected capacity change from 0 to 204800000 [ 1123.087738] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 1123.107219] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 1123.151550] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 1123.300104] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1133.607263] loop0: detected capacity change from 0 to 204800000 [ 1139.942663] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1140.024789] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1140.051234] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1141.401233] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1141.503785] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1141.630448] Lustre: lustre-MDT0000: new disk, initializing [ 1141.700471] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1141.731916] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1145.900510] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 1160.728719] loop1: detected capacity change from 0 to 204800000 [ 1166.813871] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1166.896313] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1167.209870] Lustre: lustre-OST07c6: new disk, initializing [ 1167.215199] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 1167.231815] Lustre: Skipped 1 previous similar message [ 1167.325097] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 1174.540853] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 1176.266409] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 1176.293512] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 1176.340568] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 1179.682581] Lustre: 45184:0:(osd_internal.h:1459:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 1179.704659] Lustre: 45184:0:(osd_handler.c:2096:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 1179.713211] Lustre: 45184:0:(osd_handler.c:2103:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 1179.728867] Lustre: 45184:0:(osd_handler.c:2113:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 1179.736764] Lustre: 45184:0:(osd_handler.c:2120:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 1179.749973] Lustre: 45184:0:(osd_handler.c:2127:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 1194.443827] Lustre: 45183:0:(osd_internal.h:1459:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 1194.458150] Lustre: 45183:0:(osd_handler.c:2096:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 1194.474093] Lustre: 45183:0:(osd_handler.c:2103:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 1194.480389] Lustre: 45183:0:(osd_handler.c:2113:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/4 [ 1194.489701] Lustre: 45183:0:(osd_handler.c:2120:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 1194.499374] Lustre: 45183:0:(osd_handler.c:2127:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1199.716720] Lustre: 45184:0:(osd_internal.h:1459:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 258 < left 278, rollback = 2 [ 1199.727328] Lustre: 45184:0:(osd_internal.h:1459:osd_trans_exec_op()) Skipped 1 previous similar message [ 1199.734776] Lustre: 45184:0:(osd_handler.c:2096:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 1199.739888] Lustre: 45184:0:(osd_handler.c:2096:osd_trans_dump_creds()) Skipped 1 previous similar message [ 1199.745090] Lustre: 45184:0:(osd_handler.c:2103:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 1199.751408] Lustre: 45184:0:(osd_handler.c:2103:osd_trans_dump_creds()) Skipped 1 previous similar message [ 1199.758443] Lustre: 45184:0:(osd_handler.c:2113:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/0 [ 1199.763701] Lustre: 45184:0:(osd_handler.c:2113:osd_trans_dump_creds()) Skipped 1 previous similar message [ 1199.770643] Lustre: 45184:0:(osd_handler.c:2120:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 1199.779735] Lustre: 45184:0:(osd_handler.c:2120:osd_trans_dump_creds()) Skipped 1 previous similar message [ 1199.789175] Lustre: 45184:0:(osd_handler.c:2127:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1199.798451] Lustre: 45184:0:(osd_handler.c:2127:osd_trans_dump_creds()) Skipped 1 previous similar message [ 1204.978809] Lustre: 45183:0:(osd_internal.h:1459:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 1205.000905] Lustre: 45183:0:(osd_internal.h:1459:osd_trans_exec_op()) Skipped 1 previous similar message [ 1205.005929] Lustre: 45183:0:(osd_handler.c:2096:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 1205.022360] Lustre: 45183:0:(osd_handler.c:2096:osd_trans_dump_creds()) Skipped 1 previous similar message [ 1205.029376] Lustre: 45183:0:(osd_handler.c:2103:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 1205.039743] Lustre: 45183:0:(osd_handler.c:2103:osd_trans_dump_creds()) Skipped 1 previous similar message [ 1205.051989] Lustre: 45183:0:(osd_handler.c:2113:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/0 [ 1205.064643] Lustre: 45183:0:(osd_handler.c:2113:osd_trans_dump_creds()) Skipped 1 previous similar message [ 1205.074754] Lustre: 45183:0:(osd_handler.c:2120:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 1205.082097] Lustre: 45183:0:(osd_handler.c:2120:osd_trans_dump_creds()) Skipped 1 previous similar message [ 1205.092417] Lustre: 45183:0:(osd_handler.c:2127:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1205.105218] Lustre: 45183:0:(osd_handler.c:2127:osd_trans_dump_creds()) Skipped 1 previous similar message [ 1211.634916] Lustre: 47121:0:(osd_internal.h:1459:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 1211.650773] Lustre: 47121:0:(osd_internal.h:1459:osd_trans_exec_op()) Skipped 3 previous similar messages [ 1211.665756] Lustre: 47121:0:(osd_handler.c:2096:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 1211.674618] Lustre: 47121:0:(osd_handler.c:2096:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 1211.690636] Lustre: 47121:0:(osd_handler.c:2103:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 1211.704903] Lustre: 47121:0:(osd_handler.c:2103:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 1211.720263] Lustre: 47121:0:(osd_handler.c:2113:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/0 [ 1211.733529] Lustre: 47121:0:(osd_handler.c:2113:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 1211.749484] Lustre: 47121:0:(osd_handler.c:2120:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 1211.769400] Lustre: 47121:0:(osd_handler.c:2120:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 1211.777952] Lustre: 47121:0:(osd_handler.c:2127:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1211.794866] Lustre: 47121:0:(osd_handler.c:2127:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 1221.143606] Lustre: 47121:0:(osd_internal.h:1459:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 1221.152537] Lustre: 47121:0:(osd_internal.h:1459:osd_trans_exec_op()) Skipped 3 previous similar messages [ 1221.157220] Lustre: 47121:0:(osd_handler.c:2096:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 1221.164258] Lustre: 47121:0:(osd_handler.c:2096:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 1221.169549] Lustre: 47121:0:(osd_handler.c:2103:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 1221.176266] Lustre: 47121:0:(osd_handler.c:2103:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 1221.181051] Lustre: 47121:0:(osd_handler.c:2113:osd_trans_dump_creds()) write: 2/23/0, punch: 0/0/0, quota 4/150/0 [ 1221.187143] Lustre: 47121:0:(osd_handler.c:2113:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 1221.191756] Lustre: 47121:0:(osd_handler.c:2120:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 1221.196815] Lustre: 47121:0:(osd_handler.c:2120:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 1221.202306] Lustre: 47121:0:(osd_handler.c:2127:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1221.209815] Lustre: 47121:0:(osd_handler.c:2127:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 1268.706985] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1268.714590] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1268.733263] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 1273.829018] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 1274.040224] LustreError: 47657:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 1274.294477] Lustre: server umount lustre-OST07c6 complete [ 1282.958856] LustreError: 48159:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 1282.966365] LustreError: 48159:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 1283.192763] Lustre: server umount lustre-MDT0000 complete [ 1302.031817] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [ 1304.674810] Key type lgssc unregistered [ 1305.052948] LNet: 49136:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1305.059864] LNetError: 49136:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1305.074150] LNet: Removed LNI 192.168.203.152@tcp [ 1305.869295] Key type .llcrypt unregistered [ 1305.872954] Key type ._llcrypt unregistered [ 1384.094425] Key type ._llcrypt registered [ 1384.096769] Key type .llcrypt registered [ 1384.213693] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 1385.025585] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1385.054213] alg: No test for adler32 (adler32-zlib) [ 1386.072648] Lustre: Lustre: Build Version: 2.17.0_RC4_1_g3742a8b [ 1386.286809] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [ 1388.008300] Key type lgssc registered [ 1389.097757] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1396.502664] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1396.543468] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1398.004387] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1402.344898] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 1408.483383] LustreError: 53449:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1408.514449] LustreError: 53449:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1413.621025] LustreError: 53449:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1415.268627] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1415.758502] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1420.306358] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 1426.575073] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1431.793844] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1443.454691] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1443.911451] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1450.679646] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 1456.241848] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1468.476338] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 1488.871344] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1488.875824] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1488.897400] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1488.917235] Lustre: Skipped 1 previous similar message [ 1493.691706] LustreError: 56307:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1493.789022] Lustre: server umount lustre-OST0000 complete [ 1499.110777] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1499.138451] Lustre: Skipped 1 previous similar message [ 1499.150169] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1499.163033] Lustre: Skipped 1 previous similar message [ 1504.225659] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1504.237861] Lustre: Skipped 1 previous similar message [ 1504.525685] LustreError: 56558:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1504.541758] LustreError: 56558:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1504.739597] Lustre: server umount lustre-MDT0000 complete [ 1509.021546] LustreError: 53431:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767316476 with bad export cookie 6740323810005725495 [ 1509.030089] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1509.037552] LustreError: 53431:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1509.173637] LustreError: 56811:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1509.178608] LustreError: 56811:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1509.381402] Lustre: server umount lustre-MDT0001 complete [ 1522.543721] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [ 1525.808541] Key type lgssc unregistered [ 1526.070588] LNet: 57487:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1526.075481] LNetError: 57487:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1526.098612] LNet: Removed LNI 192.168.203.152@tcp [ 1526.999354] Key type .llcrypt unregistered [ 1527.001369] Key type ._llcrypt unregistered [ 1605.121613] Key type ._llcrypt registered [ 1605.123926] Key type .llcrypt registered [ 1605.220896] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 1605.945274] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1605.997064] alg: No test for adler32 (adler32-zlib) [ 1607.050593] Lustre: Lustre: Build Version: 2.17.0_RC4_1_g3742a8b [ 1607.277284] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [ 1608.937923] Key type lgssc registered [ 1610.081153] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1618.619394] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1618.638295] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1620.286114] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1624.612675] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 1630.693556] LustreError: 61797:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1630.715027] LustreError: 61797:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1635.815254] LustreError: 61797:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1637.514092] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1638.098339] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1642.854714] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 1648.119712] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1652.075136] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1663.425596] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1663.883886] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1664.970110] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1671.503288] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 1677.776270] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1695.723797] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1695.734448] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1695.762050] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1698.787718] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1698.795410] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1698.800277] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1698.832881] Lustre: Skipped 1 previous similar message [ 1699.777128] LustreError: 64450:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1699.916246] Lustre: server umount lustre-OST0000 complete [ 1709.026855] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1709.029679] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1709.044416] Lustre: Skipped 1 previous similar message [ 1709.067230] Lustre: Skipped 1 previous similar message [ 1710.803417] LustreError: 64700:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1710.817921] LustreError: 64700:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1710.938419] Lustre: server umount lustre-MDT0000 complete [ 1714.148184] LustreError: 61798:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1714.184270] LustreError: 61798:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1715.699261] LustreError: 61779:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767316682 with bad export cookie 16734116804890366807 [ 1715.728037] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1715.838336] LustreError: 64950:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1715.847530] LustreError: 64950:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1716.053748] Lustre: server umount lustre-MDT0001 complete [ 1729.686530] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [ 1732.679630] Key type lgssc unregistered [ 1732.977203] LNet: 65627:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1732.982851] LNetError: 65627:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1733.001372] LNet: Removed LNI 192.168.203.152@tcp [ 1733.850153] Key type .llcrypt unregistered [ 1733.852543] Key type ._llcrypt unregistered [ 1822.191459] Key type ._llcrypt registered [ 1822.197258] Key type .llcrypt registered [ 1822.323936] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 1823.255985] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1823.403339] alg: No test for adler32 (adler32-zlib) [ 1824.614691] Lustre: Lustre: Build Version: 2.17.0_RC4_1_g3742a8b [ 1824.953704] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [ 1826.642327] Key type lgssc registered [ 1828.399544] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1836.165898] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1836.215268] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1837.879903] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1842.493408] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 1847.270738] LustreError: 70087:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1847.295851] LustreError: 70087:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1852.386799] LustreError: 70087:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1854.738631] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1854.960225] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1859.741143] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 1865.977040] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1871.169333] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1883.632448] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1884.142200] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1890.875342] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 1896.746156] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1907.701425] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1916.385695] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1916.397925] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1916.437495] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1917.924056] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1917.926360] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1917.938870] Lustre: Skipped 1 previous similar message [ 1917.956269] Lustre: Skipped 2 previous similar messages [ 1920.651285] LustreError: 72787:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1920.755370] Lustre: server umount lustre-MDT0000 complete [ 1923.042066] LustreError: 71246:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1923.077533] LustreError: 71246:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1925.187341] LustreError: 70067:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767316892 with bad export cookie 17029469898735578757 [ 1925.189993] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1925.196455] LustreError: 70067:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1928.161711] LustreError: 70083:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1928.162361] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1928.191869] LustreError: 70083:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1928.254114] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1928.265764] Lustre: Skipped 1 previous similar message [ 1931.432132] LustreError: 73039:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1931.453550] LustreError: 73039:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1931.709531] Lustre: server umount lustre-MDT0001 complete [ 1952.656456] LustreError: 73264:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1952.665451] LustreError: 73264:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1952.900915] Lustre: server umount lustre-OST0000 complete [ 1971.157097] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [ 1973.970426] Key type lgssc unregistered [ 1974.228657] LNet: 74343:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1974.234652] LNetError: 74343:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1974.264524] LNet: Removed LNI 192.168.203.152@tcp [ 1975.138796] Key type .llcrypt unregistered [ 1975.140539] Key type ._llcrypt unregistered [ 2054.791377] Key type ._llcrypt registered [ 2054.795113] Key type .llcrypt registered [ 2054.911988] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 2056.342227] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2056.553244] alg: No test for adler32 (adler32-zlib) [ 2057.900036] Lustre: Lustre: Build Version: 2.17.0_RC4_1_g3742a8b [ 2058.244039] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [ 2059.960192] Key type lgssc registered [ 2061.405140] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2068.858836] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2068.902316] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2070.604671] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2074.855905] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 2080.753097] LustreError: 78803:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2080.791870] LustreError: 78803:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2085.869050] LustreError: 78803:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2087.342208] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2087.699218] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2091.950636] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 2098.128895] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2101.631494] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2112.738181] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2113.390486] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2115.487698] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 2121.022796] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 2126.088796] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2136.033535] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2136.039518] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2136.071290] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2138.081489] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2138.088409] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2138.098649] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2138.114765] Lustre: Skipped 1 previous similar message [ 2141.836677] LustreError: 81436:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2141.939240] Lustre: server umount lustre-OST0000 complete [ 2160.477291] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2160.480662] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2160.491320] Lustre: Skipped 1 previous similar message [ 2160.514319] Lustre: Skipped 2 previous similar messages [ 2163.498381] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2163.501302] Lustre: Skipped 1 previous similar message [ 2166.734982] LustreError: 81862:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2166.746844] LustreError: 81862:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2166.838555] Lustre: server umount lustre-MDT0000 complete [ 2168.802766] LustreError: 80808:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2168.827039] LustreError: 80808:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2171.476559] LustreError: 78782:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767317138 with bad export cookie 14132337921038542461 [ 2171.498078] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2177.713410] LustreError: 82111:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2177.726280] LustreError: 82111:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2178.213627] Lustre: server umount lustre-MDT0001 complete [ 2192.174858] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [ 2194.775846] Key type lgssc unregistered [ 2195.029822] LNet: 82790:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2195.037723] LNetError: 82790:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2195.061810] LNet: Removed LNI 192.168.203.152@tcp [ 2195.834691] Key type .llcrypt unregistered [ 2195.836048] Key type ._llcrypt unregistered [ 2276.795731] Key type ._llcrypt registered [ 2276.802440] Key type .llcrypt registered [ 2276.914084] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 2278.183417] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2278.207927] alg: No test for adler32 (adler32-zlib) [ 2279.251039] Lustre: Lustre: Build Version: 2.17.0_RC4_1_g3742a8b [ 2279.476695] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [ 2281.144465] Key type lgssc registered [ 2282.323563] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2289.819164] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2289.836948] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2291.356724] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2295.970132] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 2300.908041] LustreError: 87264:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2300.930485] LustreError: 87264:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2306.023058] LustreError: 87264:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2308.376832] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2308.710883] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2313.410541] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 2318.774333] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2322.170295] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2332.219652] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2332.571577] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2336.783377] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 2338.872938] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 2343.923196] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2349.580560] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 2353.385312] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 2365.783281] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-01-0120h28m51s [ 2367.260462] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 2370.018233] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2370.026817] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2370.043850] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2371.107582] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2371.124326] Lustre: Skipped 4 previous similar messages [ 2372.526348] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.52@tcp (stopping) [ 2372.542195] Lustre: Skipped 1 previous similar message [ 2377.190635] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2377.195945] Lustre: Skipped 2 previous similar messages [ 2383.333619] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2383.347086] Lustre: Skipped 3 previous similar messages [ 2384.352182] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2384.468325] LustreError: 90264:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2384.662315] Lustre: server umount lustre-MDT0000 complete [ 2387.989736] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 2389.478501] LustreError: 87264:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2389.505909] LustreError: 87264:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2395.631163] LustreError: 87260:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2395.652370] LustreError: 87260:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2399.773731] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2399.958034] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2400.324371] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2400.367872] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 2401.403791] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2405.417062] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 2409.252101] Lustre: DEBUG MARKER: Wait for df (44287) ... [ 2410.885675] Lustre: DEBUG MARKER: done [ 2415.021839] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2415.023547] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2415.031797] Lustre: Skipped 3 previous similar messages [ 2415.034421] Lustre: Skipped 1 previous similar message [ 2421.111951] LustreError: 91514:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2421.114726] LustreError: 91514:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2421.240441] Lustre: server umount lustre-OST0000 complete [ 2427.361367] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2427.371724] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2431.521898] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2431.531074] Lustre: Skipped 8 previous similar messages [ 2432.184275] LustreError: 91764:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2432.189803] LustreError: 91764:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2432.354683] Lustre: server umount lustre-MDT0000 complete [ 2437.461839] LustreError: 89946:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767317404 with bad export cookie 6520141523108849768 [ 2437.470581] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2437.600076] LustreError: 92013:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2437.607794] LustreError: 92013:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2437.733566] Lustre: server umount lustre-MDT0001 complete [ 2453.059490] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [ 2455.791747] Key type lgssc unregistered [ 2456.050671] LNet: 92691:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2456.057582] LNetError: 92691:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2456.072519] LNet: Removed LNI 192.168.203.152@tcp [ 2456.864284] Key type .llcrypt unregistered [ 2456.869879] Key type ._llcrypt unregistered [ 2515.978431] Key type ._llcrypt registered [ 2515.981180] Key type .llcrypt registered [ 2516.058977] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_hostid [ 2583.103321] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 2583.997297] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2584.105720] alg: No test for adler32 (adler32-zlib) [ 2585.198462] Lustre: Lustre: Build Version: 2.17.0_RC4_1_g3742a8b [ 2585.384802] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [ 2587.104165] Key type lgssc registered [ 2588.252769] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2607.220411] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2619.329256] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2628.160870] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2637.019714] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2652.740118] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2652.821679] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2652.835950] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2654.081733] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2654.109389] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2654.238231] Lustre: lustre-MDT0000: new disk, initializing [ 2654.325080] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2654.340529] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2659.110904] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 2675.092829] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2675.182310] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2675.312092] Lustre: 102527:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2675.354969] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2675.358382] Lustre: Skipped 1 previous similar message [ 2675.418105] Lustre: lustre-MDT0001: new disk, initializing [ 2675.522707] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2675.578657] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2675.597846] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2679.594681] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 2687.057747] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2690.339828] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2702.488791] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2702.591832] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2702.856276] Lustre: lustre-OST0000: new disk, initializing [ 2702.863753] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2702.968689] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2708.607955] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2708.617379] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2708.722147] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 2708.766215] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2715.659448] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2719.702760] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2719.893264] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2724.172706] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2724.371783] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2729.442134] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2729.444138] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2729.454422] LustreError: Skipped 1 previous similar message [ 2729.465320] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2732.179886] LustreError: 106160:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2732.427734] Lustre: server umount lustre-OST0000 complete [ 2739.681098] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2739.681609] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2739.690297] Lustre: Skipped 2 previous similar messages [ 2739.705985] Lustre: Skipped 2 previous similar messages [ 2743.406203] LustreError: 106411:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2743.410762] LustreError: 106411:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2743.681519] Lustre: server umount lustre-MDT0000 complete [ 2747.454862] LustreError: 102522:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767317714 with bad export cookie 9983782161990342511 [ 2747.460683] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2747.461410] LustreError: 102522:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2747.587415] LustreError: 106660:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2747.594770] LustreError: 106660:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2747.754646] Lustre: server umount lustre-MDT0001 complete [ 2755.105625] Lustre: DEBUG MARKER: == conf-sanity test 35b: Continue reconnection retries, if the active server is busy ========================================================== 20:35:21 (1767317721) [ 2765.025429] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2765.339856] LustreError: 107526:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2765.355115] LustreError: 107526:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2765.412794] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2768.847698] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 2770.403326] LustreError: 107526:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2775.524915] LustreError: 107526:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2780.646241] LustreError: 107526:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2780.748884] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2781.057592] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2786.016327] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 2791.561115] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2796.232948] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2806.905909] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2807.309826] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2813.160364] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 2818.862271] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2829.433860] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-01-0120h36m35s [ 2830.959903] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 2838.137212] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 2840.615265] Lustre: DEBUG MARKER: Creating a test file and stat it [ 2840.678310] LustreError: 107522:0:(mdt_handler.c:2334:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 2860.940945] Lustre: lustre-MDT0000: Client a0148a0e-75b5-473f-80d4-5d76eabeff6b (at 192.168.203.52@tcp) reconnecting [ 2880.688121] LustreError: 107522:0:(mdt_handler.c:2334:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 2882.894445] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 2885.582986] Lustre: DEBUG MARKER: done [ 2892.264925] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2892.277872] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2892.318717] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2893.686648] LustreError: 110919:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2893.696212] LustreError: 110919:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2893.837482] Lustre: server umount lustre-OST0000 complete [ 2903.523371] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2903.536091] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2903.538302] Lustre: Skipped 2 previous similar messages [ 2903.545820] Lustre: Skipped 2 previous similar messages [ 2905.264808] LustreError: 111168:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2905.281163] LustreError: 111168:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2905.391399] Lustre: server umount lustre-MDT0000 complete [ 2910.254961] LustreError: 110303:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767317877 with bad export cookie 9983782161990343393 [ 2910.264608] LustreError: 110303:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2910.275337] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2910.723760] Lustre: server umount lustre-MDT0001 complete [ 2924.740158] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [ 2927.662867] Key type lgssc unregistered [ 2928.022721] LNet: 112094:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2928.030818] LNetError: 112094:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2928.065407] LNet: Removed LNI 192.168.203.152@tcp [ 2928.851652] Key type .llcrypt unregistered [ 2928.854313] Key type ._llcrypt unregistered [ 2989.326535] Key type ._llcrypt registered [ 2989.329407] Key type .llcrypt registered [ 2989.446367] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_hostid [ 3065.495895] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 3066.715951] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3066.726744] alg: No test for adler32 (adler32-zlib) [ 3067.770645] Lustre: Lustre: Build Version: 2.17.0_RC4_1_g3742a8b [ 3068.073556] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [ 3069.772533] Key type lgssc registered [ 3071.115989] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3092.104070] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3104.132499] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3114.113552] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3125.405726] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3143.382581] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3143.457970] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3143.483416] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3144.733472] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3144.770970] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3144.854699] Lustre: lustre-MDT0000: new disk, initializing [ 3144.985670] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3145.023238] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3149.097713] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 3167.296748] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3167.374805] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3167.449269] Lustre: 121940:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3167.496978] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3167.500393] Lustre: Skipped 1 previous similar message [ 3167.579779] Lustre: lustre-MDT0001: new disk, initializing [ 3167.692251] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3167.756495] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3167.776898] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3172.374439] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 3180.574262] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3184.643234] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3198.787495] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3198.856886] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3199.133322] Lustre: lustre-OST0000: new disk, initializing [ 3199.145233] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3199.255904] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3204.704281] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3204.711093] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3204.897494] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3205.251716] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 3212.862959] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3217.078227] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3217.241621] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3221.712640] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3221.969577] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3225.572175] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3225.599702] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3225.635643] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3230.689434] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3230.714135] Lustre: Skipped 2 previous similar messages [ 3235.822812] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3235.832423] Lustre: Skipped 1 previous similar message [ 3238.368876] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3238.403672] LustreError: 125574:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3238.648467] Lustre: server umount lustre-OST0000 complete [ 3244.530102] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3244.534626] LustreError: Skipped 1 previous similar message [ 3244.551864] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3244.571275] Lustre: Skipped 1 previous similar message [ 3244.589169] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3246.051901] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3249.810171] LustreError: 125826:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3249.815066] LustreError: 125826:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3249.939034] Lustre: server umount lustre-MDT0000 complete [ 3255.799397] LustreError: 121935:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767318222 with bad export cookie 819181276666409522 [ 3255.832409] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3256.045395] LustreError: 126075:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3256.059056] LustreError: 126075:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3256.267329] Lustre: server umount lustre-MDT0001 complete [ 3267.743599] Lustre: DEBUG MARKER: == conf-sanity test 36: df report consistency on OSTs with different block size ========================================================== 20:43:53 (1767318233) [ 3269.765357] Lustre: DEBUG MARKER: SKIP: conf-sanity test_36 remote OST [ 3271.461679] Lustre: DEBUG MARKER: == conf-sanity test 37: verify set tunables works for symlink device ========================================================== 20:43:57 (1767318237) [ 3349.492412] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 3354.942667] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3355.452576] LustreError: 130063:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3355.477594] LustreError: 130063:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3355.547899] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3356.542425] Lustre: Failing over lustre-MDT0000 [ 3356.548332] LustreError: 130091:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 3356.559777] LustreError: 130091:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 3356.568850] LustreError: 130091:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 1, retries 0, failed: rc = -5 [ 3356.732866] LustreError: 130142:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3356.743795] LustreError: 130142:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3356.923280] Lustre: server umount lustre-MDT0000 complete [ 3363.153423] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 20:45:29 (1767318329) [ 3373.745084] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3374.211916] LustreError: 130903:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3374.269378] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3378.515518] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 3379.683127] LustreError: 130903:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3384.806099] LustreError: 130903:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3389.926896] LustreError: 130903:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3390.463524] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3390.844517] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3395.275458] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 3401.062780] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3404.751304] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3414.357186] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3414.741582] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3421.480605] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 3426.948347] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3432.199252] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 3440.610380] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3440.629505] Lustre: Skipped 1 previous similar message [ 3440.635538] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3440.646394] Lustre: Skipped 4 previous similar messages [ 3444.940612] LustreError: 133683:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3444.943497] LustreError: 133683:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3445.006968] Lustre: server umount lustre-MDT0000 complete [ 3445.737357] LustreError: 130903:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3445.755034] LustreError: 130903:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3450.814783] LustreError: 130885:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767318417 with bad export cookie 819181276666410677 [ 3450.827985] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3450.833904] LustreError: 130885:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 3450.872369] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3450.891089] Lustre: Skipped 1 previous similar message [ 3450.928373] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3451.217903] Lustre: server umount lustre-MDT0001 complete [ 3455.482354] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 3461.951763] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3475.257676] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3476.448118] LustreError: 135065:0:(import.c:337:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [ 3476.449050] LustreError: 118402:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000091027d03 x1853167275948288/t0(0) o250->MGC192.168.203.152@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3476.883510] LustreError: 132901:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3476.911411] LustreError: 132901:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3477.031696] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3477.151704] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 3481.131522] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3481.251992] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 3494.461365] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3499.838947] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 3500.018271] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3505.559446] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3509.628436] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3520.497851] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3520.499714] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3520.506189] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3520.524148] Lustre: Skipped 1 previous similar message [ 3523.185728] LustreError: 136723:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3523.200927] LustreError: 136723:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 3523.424183] Lustre: server umount lustre-MDT0000 complete [ 3525.606483] LustreError: 135077:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3525.651508] LustreError: 135077:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 3528.051949] LustreError: 130885:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767318495 with bad export cookie 819181276666412581 [ 3528.060706] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3528.062955] LustreError: 130885:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3528.323049] Lustre: server umount lustre-MDT0001 complete [ 3536.146258] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3546.850101] Lustre: 118406:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767318497/real 1767318497] req@0000000065cb460d x1853167275974912/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1767318513 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3546.894232] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3546.911657] Lustre: Skipped 1 previous similar message [ 3549.159127] Lustre: 118404:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767318500/real 1767318500] req@00000000cff511ed x1853167275975040/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1767318516 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3549.429630] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3551.264210] Lustre: 118404:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767318502/real 1767318502] req@000000000aa7c1fc x1853167275975168/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1767318518 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3553.057520] LustreError: 118402:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000bdabf76d x1853167275975936/t0(0) o250->MGC192.168.203.152@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3553.089901] LustreError: 118402:0:(client.c:1390:ptlrpc_import_delay_req()) Skipped 1 previous similar message [ 3553.559038] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3553.564117] Lustre: Skipped 1 previous similar message [ 3553.632456] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 3554.208573] Lustre: 118405:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767318505/real 1767318505] req@00000000361b3dc9 x1853167275975424/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1767318521 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3557.203926] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 3560.744921] Lustre: 118405:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767318511/real 1767318511] req@000000003f327038 x1853167275975552/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1767318527 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3569.039902] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3573.247224] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 3574.767107] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3578.886125] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3583.481238] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3595.233656] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3595.237616] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3595.251689] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3595.257020] Lustre: Skipped 2 previous similar messages [ 3598.181337] LustreError: 139758:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3598.204533] LustreError: 139758:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 3598.834810] Lustre: server umount lustre-MDT0000 complete [ 3601.384302] LustreError: 138118:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3601.396672] LustreError: 138118:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 10 previous similar messages [ 3604.456564] LustreError: 130884:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767318571 with bad export cookie 819181276666413953 [ 3604.463611] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3604.467899] LustreError: 130884:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3604.861084] Lustre: server umount lustre-MDT0001 complete [ 3611.843367] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3616.246292] Lustre: DEBUG MARKER: files compared the same [ 3628.574685] Lustre: server umount lustre-OST0000 complete [ 3655.114934] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [ 3657.165977] Key type lgssc unregistered [ 3657.471575] LNet: 141778:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3657.482764] LNetError: 141778:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3657.522571] LNet: Removed LNI 192.168.203.152@tcp [ 3658.779243] Key type .llcrypt unregistered [ 3658.784163] Key type ._llcrypt unregistered [ 3746.351576] Key type ._llcrypt registered [ 3746.358694] Key type .llcrypt registered [ 3746.462304] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 3747.720595] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3747.830433] alg: No test for adler32 (adler32-zlib) [ 3748.954353] Lustre: Lustre: Build Version: 2.17.0_RC4_1_g3742a8b [ 3749.178820] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [ 3750.928189] Key type lgssc registered [ 3752.230227] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3760.313625] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3760.331934] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3761.811739] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3766.879472] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug +malloc all [ 3771.372621] LustreError: 146239:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3771.385605] LustreError: 146239:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3776.483734] LustreError: 146239:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3779.448334] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3779.861381] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3784.661763] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug +malloc all [ 3790.068576] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3794.522153] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3804.925992] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3805.317539] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3806.390482] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 3811.182609] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug +malloc all [ 3816.644432] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3835.374779] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3835.378431] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3835.404641] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3840.261131] LustreError: 148891:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3840.535220] Lustre: server umount lustre-OST0000 complete [ 3846.626073] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3846.630187] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3846.639775] Lustre: Skipped 1 previous similar message [ 3846.648191] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3846.657565] Lustre: Skipped 1 previous similar message [ 3850.722861] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3850.723633] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3852.027684] LustreError: 149141:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3852.043972] LustreError: 149141:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3852.324947] Lustre: server umount lustre-MDT0000 complete [ 3854.887198] LustreError: 146235:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3854.922716] LustreError: 146235:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3858.506157] LustreError: 146221:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767318825 with bad export cookie 13043453694130057315 [ 3858.514038] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3858.539027] LustreError: 146221:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3858.797939] LustreError: 149390:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3858.807783] LustreError: 149390:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3859.304944] Lustre: server umount lustre-MDT0001 complete [ 3875.538306] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [ 3878.842903] Key type lgssc unregistered [ 3879.109506] LNet: 150068:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3879.119591] LNetError: 150068:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3879.160806] LNet: Removed LNI 192.168.203.152@tcp [ 3880.160705] Key type .llcrypt unregistered [ 3880.166302] Key type ._llcrypt unregistered [ 3969.749940] Key type ._llcrypt registered [ 3969.752677] Key type .llcrypt registered [ 3969.848246] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 3971.380223] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3971.396870] alg: No test for adler32 (adler32-zlib) [ 3972.558768] Lustre: Lustre: Build Version: 2.17.0_RC4_1_g3742a8b [ 3972.752445] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [ 3974.417502] Key type lgssc registered [ 3975.711752] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3983.758979] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3983.785165] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3999.393668] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4004.512334] LustreError: 154501:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.152@tcp: failed processing log, type 4: rc = -110 [ 4031.264907] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4037.796131] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 4044.342666] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4058.345201] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4058.602661] Lustre: *** cfs_fail_loc=706, val=0*** [ 4059.194899] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4059.296520] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 4064.251153] LustreError: 154527:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4064.312758] LustreError: 154527:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4065.133484] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 4066.348336] LustreError: 154527:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4066.399617] LustreError: 154527:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4071.404455] LustreError: 154527:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4071.423860] LustreError: 154527:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4076.513946] LustreError: 155667:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4076.549683] LustreError: 155667:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4081.339983] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4081.774650] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4087.861174] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 4093.377306] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4097.241323] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4100.065763] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4100.077468] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4100.103114] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4102.114984] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4102.123978] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4102.153219] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4105.345247] LustreError: 157215:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 4105.559485] Lustre: server umount lustre-OST0000 complete [ 4112.353409] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4112.356877] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4112.384550] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4112.388661] Lustre: Skipped 1 previous similar message [ 4116.297243] LustreError: 157465:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 4116.318327] LustreError: 157465:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4116.516785] Lustre: server umount lustre-MDT0000 complete [ 4117.985391] LustreError: 155667:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4118.009670] LustreError: 155667:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4121.352215] LustreError: 154509:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767319088 with bad export cookie 3111129835333217404 [ 4121.357165] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4121.369971] LustreError: 154509:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4121.520913] LustreError: 157715:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4121.527679] LustreError: 157715:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4121.726006] Lustre: server umount lustre-MDT0001 complete [ 4136.002833] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [ 4138.853910] Key type lgssc unregistered [ 4139.221926] LNet: 158392:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4139.233404] LNetError: 158392:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4139.248129] LNet: Removed LNI 192.168.203.152@tcp [ 4140.206320] Key type .llcrypt unregistered [ 4140.209167] Key type ._llcrypt unregistered [ 4228.921771] Key type ._llcrypt registered [ 4228.923394] Key type .llcrypt registered [ 4229.117766] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 4230.547378] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4230.577407] alg: No test for adler32 (adler32-zlib) [ 4231.678682] Lustre: Lustre: Build Version: 2.17.0_RC4_1_g3742a8b [ 4231.925866] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [ 4233.608235] Key type lgssc registered [ 4234.928696] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4242.693770] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4242.725749] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4247.553586] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 4257.458467] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4258.134538] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4262.599283] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 4263.392976] LustreError: 163906:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4263.420616] LustreError: 163906:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4268.513794] LustreError: 163906:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4268.536965] LustreError: 163906:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4273.635621] LustreError: 163910:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4273.654703] LustreError: 163910:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4275.202060] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4275.557224] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4280.815877] LustreError: 163907:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4280.851688] LustreError: 163907:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 4282.219286] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 4288.996023] LustreError: 163910:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4289.017101] LustreError: 163910:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 4294.832831] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4294.844777] Lustre: MGS: Client 0ed421a4-11bb-4ca5-9169-238ec3a833d7 (at 0@lo) reconnecting [ 4295.154184] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4295.161966] LustreError: 163921:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 4295.250515] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 4299.467359] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 4315.622273] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4315.642100] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4315.645557] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4319.686334] LustreError: 166287:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 4319.907103] Lustre: server umount lustre-OST0000 complete [ 4325.864351] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4325.872438] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4325.877126] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4325.877133] Lustre: Skipped 1 previous similar message [ 4325.881774] Lustre: Skipped 2 previous similar messages [ 4330.746641] LustreError: 166536:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4330.753463] LustreError: 166536:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4330.788665] Lustre: server umount lustre-MDT0000 complete [ 4330.979217] LustreError: 165818:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4331.000154] LustreError: 165818:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 4335.281989] LustreError: 166786:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4335.291450] LustreError: 166786:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4335.489196] Lustre: server umount lustre-MDT0001 complete [ 4339.666429] LustreError: 167037:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 4339.674408] LustreError: 167037:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4339.728928] Lustre: server umount MGS complete [ 4351.596862] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 21:01:57 (1767319317) [ 4411.442938] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_hostid [ 4480.896701] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 4503.062501] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4516.122266] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4525.766293] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4536.295584] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4551.671860] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4551.746605] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4557.166345] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 4573.826035] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4573.903250] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4574.157699] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 4574.214374] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4574.311201] Lustre: lustre-MDT0001: new disk, initializing [ 4574.423454] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4574.455484] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 4574.458463] Lustre: 178053:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 4574.466784] LustreError: 178053:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 4580.079753] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 4600.601617] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4600.726275] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4601.037296] Lustre: lustre-OST0000: new disk, initializing [ 4601.044199] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4601.145698] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4608.988585] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 4612.195478] Lustre: 179594:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@00000000ecdcf9db x1853168496454016/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1767319595 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 4617.458351] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4628.672427] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4628.685786] Lustre: MGS: Client 57be89fb-7422-4a6d-9811-89ae12b7581f (at 0@lo) reconnecting [ 4628.719743] Lustre: 176991:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4628.737126] LustreError: 176991:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x0 : rc = -11 [ 4628.761535] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4628.837757] Lustre: lustre-MDT0000: new disk, initializing [ 4629.021320] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4629.053993] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4629.085296] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 4630.118332] Lustre: 180307:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@000000000c18f31d x1853168496460544/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1767319613 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 4633.663247] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 4635.237843] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 4639.217857] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4639.234277] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4639.353821] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 4668.394357] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4668.400641] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4668.418660] Lustre: Skipped 1 previous similar message [ 4668.434875] Lustre: Skipped 2 previous similar messages [ 4671.642079] LustreError: 180829:0:(obd_class.h:479:obd_check_dev()) Device 11 not setup [ 4671.652812] LustreError: 180829:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 4671.906705] Lustre: server umount lustre-OST0000 complete [ 4678.632905] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4678.636569] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4678.643338] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4678.648657] Lustre: Skipped 1 previous similar message [ 4683.220628] LustreError: 181079:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 4683.229209] LustreError: 181079:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4683.253320] Lustre: server umount lustre-MDT0000 complete [ 4683.746379] LustreError: 178028:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4683.761613] LustreError: 178028:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4702.176159] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4702.272677] LustreError: 181331:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 4702.285891] LustreError: 181331:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4702.501848] Lustre: server umount lustre-MDT0001 complete [ 4708.071385] Lustre: server umount MGS complete [ 4723.789303] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 21:08:09 (1767319689) [ 4750.842542] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [ 4753.566430] Key type lgssc unregistered [ 4753.871312] LNet: 183207:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4753.889925] LNetError: 183207:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4753.922349] LNet: Removed LNI 192.168.203.152@tcp [ 4754.972956] Key type .llcrypt unregistered [ 4754.975070] Key type ._llcrypt unregistered [ 4833.166042] Key type ._llcrypt registered [ 4833.167983] Key type .llcrypt registered [ 4833.342684] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 4834.841989] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4835.132774] alg: No test for adler32 (adler32-zlib) [ 4836.451372] Lustre: Lustre: Build Version: 2.17.0_RC4_1_g3742a8b [ 4836.880601] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [ 4838.609377] Key type lgssc registered [ 4840.340341] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4850.425136] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4850.460940] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4852.156804] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4857.030373] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 4862.446273] LustreError: 187473:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4862.468740] LustreError: 187473:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4867.553372] LustreError: 187473:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4869.083046] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4869.494922] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4874.856943] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 4880.347545] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4884.720300] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4895.849627] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4896.232268] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4897.261804] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 4903.232485] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 4909.222657] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4922.582133] LustreError: 190106:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4922.722539] Lustre: server umount lustre-OST0000 complete [ 4930.019492] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4930.021718] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4930.050604] Lustre: Skipped 1 previous similar message [ 4930.073056] Lustre: Skipped 1 previous similar message [ 4935.140412] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4935.160966] Lustre: Skipped 1 previous similar message [ 4937.193253] LustreError: 190356:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4937.204315] LustreError: 190356:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4937.328251] Lustre: server umount lustre-MDT0000 complete [ 4940.259758] LustreError: 187473:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4940.290870] LustreError: 187473:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4941.282534] Lustre: 186782:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767319892/real 1767319892] req@000000003a51ad1e x1853169129779584/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1767319908 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4941.300610] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4941.992178] LustreError: 187455:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767319909 with bad export cookie 10111808030343426238 [ 4941.993087] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4942.010359] LustreError: 187455:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4942.170114] LustreError: 190606:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 4942.178792] LustreError: 190606:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4942.470014] Lustre: server umount lustre-MDT0001 complete [ 4957.434566] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [ 4961.107546] Key type lgssc unregistered [ 4961.505594] LNet: 191283:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4961.513199] LNetError: 191283:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4961.552281] LNet: Removed LNI 192.168.203.152@tcp [ 4962.551551] Key type .llcrypt unregistered [ 4962.553881] Key type ._llcrypt unregistered [ 5039.109722] Key type ._llcrypt registered [ 5039.111724] Key type .llcrypt registered [ 5039.230769] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 5040.491231] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5040.523238] alg: No test for adler32 (adler32-zlib) [ 5041.688355] Lustre: Lustre: Build Version: 2.17.0_RC4_1_g3742a8b [ 5041.899623] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [ 5043.624225] Key type lgssc registered [ 5044.948193] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5053.019181] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5053.023800] LustreError: 195268:0:(tgt_mount.c:2435:server_fill_super()) cfs_race id 716 sleeping [ 5053.045330] LustreError: 195279:0:(tgt_mount.c:2435:server_fill_super()) cfs_fail_race id 716 waking [ 5053.051506] LustreError: 195268:0:(tgt_mount.c:2435:server_fill_super()) cfs_fail_race id 716 awake: rc=4977 [ 5053.058981] LustreError: 195268:0:(tgt_mount.c:2447:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 5053.062492] LustreError: 195268:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 5053.070290] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5054.587656] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5064.163931] LustreError: 195318:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5064.180639] LustreError: 195318:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5066.643757] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5066.886564] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5071.321573] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 5077.173966] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing lsmod [ 5081.161965] LustreError: 196697:0:(tgt_mount.c:2435:server_fill_super()) cfs_race id 716 sleeping [ 5081.166624] LustreError: 196696:0:(tgt_mount.c:2435:server_fill_super()) cfs_fail_race id 716 waking [ 5081.187458] LustreError: 196697:0:(tgt_mount.c:2435:server_fill_super()) cfs_fail_race id 716 awake: rc=4982 [ 5081.203767] LustreError: 196697:0:(tgt_mount.c:2447:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 5081.213720] LustreError: 196697:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 5081.219261] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5081.737412] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5087.204041] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5087.205572] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5087.214318] Lustre: Skipped 2 previous similar messages [ 5087.229557] Lustre: Skipped 2 previous similar messages [ 5091.365985] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5091.372242] Lustre: Skipped 2 previous similar messages [ 5091.563953] LustreError: 196933:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5091.834038] Lustre: server umount lustre-MDT0000 complete [ 5096.419219] LustreError: 195312:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5096.433889] LustreError: 195312:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5096.712825] LustreError: 195294:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767320063 with bad export cookie 10769577013847638753 [ 5096.727580] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5096.731450] LustreError: 195294:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5097.569423] LustreError: 195944:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000008c2634b6 x1853169345778304/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5097.626049] LustreError: 197185:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 5097.635648] LustreError: 197185:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5097.759897] Lustre: server umount lustre-MDT0001 complete [ 5111.813811] LustreError: 197387:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 5111.816870] LustreError: 197387:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5111.928457] Lustre: server umount lustre-OST0000 complete [ 5124.480166] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5125.127253] LustreError: 198057:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5125.229561] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5129.768961] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 5130.722588] LustreError: 198057:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5135.844827] LustreError: 198057:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5142.072475] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5142.471488] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5147.206887] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 5152.861082] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5157.766104] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5168.699101] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5169.047786] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5170.105345] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 5176.203877] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 5182.159474] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5194.741034] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 5194.746453] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5194.772376] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5197.795422] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5197.804117] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5197.804722] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5197.840260] Lustre: Skipped 1 previous similar message [ 5199.740301] LustreError: 200693:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5199.749496] LustreError: 200693:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5199.876885] Lustre: server umount lustre-OST0000 complete [ 5208.034742] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5208.068089] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5208.080978] Lustre: Skipped 1 previous similar message [ 5218.275114] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5218.287019] Lustre: Skipped 3 previous similar messages [ 5219.296459] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5219.461129] LustreError: 200944:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5219.469534] LustreError: 200944:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5219.557111] Lustre: server umount lustre-MDT0000 complete [ 5223.394202] LustreError: 200606:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5223.407172] LustreError: 200606:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5224.293109] LustreError: 198036:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767320191 with bad export cookie 10769577013847639453 [ 5224.294727] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5224.301077] LustreError: 198036:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5224.664975] Lustre: server umount lustre-MDT0001 complete [ 5242.345842] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [ 5245.581741] Key type lgssc unregistered [ 5245.912653] LNet: 201872:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5245.918945] LNetError: 201872:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5245.940664] LNet: Removed LNI 192.168.203.152@tcp [ 5247.133607] Key type .llcrypt unregistered [ 5247.135749] Key type ._llcrypt unregistered [ 5336.532657] Key type ._llcrypt registered [ 5336.534478] Key type .llcrypt registered [ 5336.624139] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 5337.877908] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5338.020579] alg: No test for adler32 (adler32-zlib) [ 5339.119933] Lustre: Lustre: Build Version: 2.17.0_RC4_1_g3742a8b [ 5339.400806] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [ 5341.208929] Key type lgssc registered [ 5342.769960] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5352.658979] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5352.688867] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5354.334225] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5359.190463] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 5364.712204] LustreError: 206333:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5364.739444] LustreError: 206333:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5369.839241] LustreError: 206333:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5371.805513] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5372.428368] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5378.810663] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 5385.694326] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5389.321783] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5400.179915] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5400.542573] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5406.592548] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 5413.332935] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5424.111948] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 5433.701236] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 5438.321711] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 5444.580577] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5444.592190] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5444.595863] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5444.607379] Lustre: Skipped 1 previous similar message [ 5444.648965] Lustre: Skipped 1 previous similar message [ 5449.599144] LustreError: 209097:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5449.713558] LustreError: 208320:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5449.828995] Lustre: server umount lustre-OST0000 complete [ 5459.425759] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5459.440985] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5459.452241] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5462.585865] LustreError: 209348:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5462.593749] LustreError: 209348:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5462.781924] Lustre: server umount lustre-MDT0000 complete [ 5464.041188] LustreError: 207420:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5464.058596] LustreError: 207420:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5468.504570] LustreError: 208943:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767320435 with bad export cookie 10031307595593127699 [ 5468.505586] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5468.520294] LustreError: 208943:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5468.685336] LustreError: 209598:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 5468.688594] LustreError: 209598:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5468.860975] Lustre: server umount lustre-MDT0001 complete [ 5483.776247] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [ 5486.325899] Key type lgssc unregistered [ 5486.564951] LNet: 210275:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5486.588967] LNetError: 210275:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5486.616819] LNet: Removed LNI 192.168.203.152@tcp [ 5487.485256] Key type .llcrypt unregistered [ 5487.486614] Key type ._llcrypt unregistered [ 5565.769762] Key type ._llcrypt registered [ 5565.773159] Key type .llcrypt registered [ 5565.910706] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 5566.936628] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5567.053060] alg: No test for adler32 (adler32-zlib) [ 5568.087526] Lustre: Lustre: Build Version: 2.17.0_RC4_1_g3742a8b [ 5568.317770] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [ 5570.032135] Key type lgssc registered [ 5571.191812] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5578.203621] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5578.287861] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5579.907456] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5584.199659] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 5589.474915] LustreError: 214538:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5589.493240] LustreError: 214538:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5594.600426] LustreError: 214538:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5596.294803] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5597.023893] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5601.706113] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 5607.426627] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5610.931104] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5622.547635] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5622.920411] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5623.941928] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 5629.242895] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 5634.255604] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5647.849181] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 5647.858547] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5647.871865] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5651.943328] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5651.949349] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5651.979887] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5653.754164] LustreError: 217190:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5653.826717] Lustre: server umount lustre-OST0000 complete [ 5662.178574] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5662.187803] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5662.205729] Lustre: Skipped 1 previous similar message [ 5662.219338] Lustre: Skipped 1 previous similar message [ 5664.961302] LustreError: 217441:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5664.973908] LustreError: 217441:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5665.328298] Lustre: server umount lustre-MDT0000 complete [ 5667.299107] LustreError: 214534:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5667.319153] LustreError: 214534:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5669.983391] LustreError: 214518:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767320637 with bad export cookie 181621304102811020 [ 5669.995805] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5670.149114] LustreError: 217690:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 5670.161250] LustreError: 217690:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5670.817643] Lustre: server umount lustre-MDT0001 complete [ 5685.418854] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [ 5688.279746] Key type lgssc unregistered [ 5688.544152] LNet: 218369:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5688.552551] LNetError: 218369:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5688.571623] LNet: Removed LNI 192.168.203.152@tcp [ 5689.414418] Key type .llcrypt unregistered [ 5689.417402] Key type ._llcrypt unregistered [ 5768.105967] Key type ._llcrypt registered [ 5768.111680] Key type .llcrypt registered [ 5768.311406] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 5769.680098] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5769.811867] alg: No test for adler32 (adler32-zlib) [ 5771.049570] Lustre: Lustre: Build Version: 2.17.0_RC4_1_g3742a8b [ 5771.344949] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [ 5773.049732] Key type lgssc registered [ 5774.476792] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5781.572256] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5781.594214] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5783.223987] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5787.980705] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 5793.762276] LustreError: 222828:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5793.772622] LustreError: 222828:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5798.887866] LustreError: 222828:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5800.156923] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5800.500363] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5804.869801] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 5810.113670] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5814.598154] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5825.198767] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5825.621834] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5828.740099] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 5831.305452] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 5836.455149] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5851.408030] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 5854.920545] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 5854.927830] Lustre: Skipped 2 previous similar messages [ 5859.291734] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 5859.300137] Lustre: Skipped 2 previous similar messages [ 5859.940117] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 5859.966285] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 5868.261493] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.203.52@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 5868.266842] Lustre: Skipped 2 previous similar messages [ 5869.959989] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.203.52@tcp 192.168.0.[2,10]@tcp [ 5874.987400] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.203.52/24@tcp in log lustre-MDT0000 [ 5874.995501] Lustre: Skipped 2 previous similar messages [ 5876.259352] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.52/24@tcp [ 5876.264858] Lustre: Skipped 1 previous similar message [ 5883.553189] LustreError: 226594:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5883.634282] Lustre: server umount lustre-OST0000 complete [ 5891.556951] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5891.559075] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5891.577421] Lustre: Skipped 1 previous similar message [ 5891.614084] Lustre: Skipped 1 previous similar message [ 5896.675212] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5896.682301] Lustre: Skipped 1 previous similar message [ 5898.746050] LustreError: 226843:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5898.759972] LustreError: 226843:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5898.890696] Lustre: server umount lustre-MDT0000 complete [ 5901.800500] LustreError: 225373:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5901.823981] LustreError: 225373:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5902.295201] Lustre: 222138:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767320853/real 1767320853] req@0000000097e5cd0e x1853170110228352/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1767320869 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5902.338743] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5902.816207] Lustre: 222136:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767320853/real 1767320853] req@00000000e470f9fb x1853170110227968/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1767320869 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5903.007349] LustreError: 222808:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767320870 with bad export cookie 5947847879676324218 [ 5903.009138] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5903.020736] LustreError: 222808:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5903.130958] LustreError: 227093:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 5903.134482] LustreError: 227093:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5903.299172] Lustre: server umount lustre-MDT0001 complete [ 5918.749977] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [ 5922.056402] Key type lgssc unregistered [ 5922.299580] LNet: 227770:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5922.310168] LNetError: 227770:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5923.381634] LNet: Removed LNI 192.168.203.152@tcp [ 5924.354584] Key type .llcrypt unregistered [ 5924.360814] Key type ._llcrypt unregistered [ 6005.205735] Key type ._llcrypt registered [ 6005.209524] Key type .llcrypt registered [ 6005.280424] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 6006.320406] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6006.368440] alg: No test for adler32 (adler32-zlib) [ 6007.476165] Lustre: Lustre: Build Version: 2.17.0_RC4_1_g3742a8b [ 6007.655248] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [ 6009.328195] Key type lgssc registered [ 6010.442871] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6018.408631] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6018.424739] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6019.952626] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 6019.999052] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.203.52/24@tcp [ 6020.028156] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6024.102616] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 6030.311108] LustreError: 232423:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6030.333307] LustreError: 232423:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6034.871369] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6035.287221] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 6035.308587] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.52/24@tcp [ 6035.385748] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6039.650572] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 6045.434873] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6049.136988] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6060.058550] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6060.644640] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6065.772614] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 6067.351860] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 6073.493771] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6086.630910] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6086.632776] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6086.640240] LustreError: Skipped 1 previous similar message [ 6086.665430] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6090.726657] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6090.738486] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6090.740018] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6090.765326] Lustre: Skipped 1 previous similar message [ 6092.441475] LustreError: 235108:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6092.742509] Lustre: server umount lustre-OST0000 complete [ 6100.967872] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6100.970145] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6100.980868] Lustre: Skipped 1 previous similar message [ 6101.000460] Lustre: Skipped 1 previous similar message [ 6103.615841] LustreError: 235361:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6103.630092] LustreError: 235361:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6105.766541] LustreError: 235361:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 6105.771293] LustreError: 235361:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6105.804656] Lustre: server umount lustre-MDT0000 complete [ 6106.085442] LustreError: 232422:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6106.087151] LustreError: 232417:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6110.178000] LustreError: 232417:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6110.209106] LustreError: 232417:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6110.517325] LustreError: 232402:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767321077 with bad export cookie 17564635747839456077 [ 6110.533371] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6110.659560] LustreError: 235611:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 6110.668869] LustreError: 235611:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 6110.860805] Lustre: server umount lustre-MDT0001 complete [ 6125.984231] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [ 6129.687869] Key type lgssc unregistered [ 6130.138836] LNet: 236285:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6130.155377] LNetError: 236285:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6131.178880] LNet: Removed LNI 192.168.203.152@tcp [ 6132.377697] Key type .llcrypt unregistered [ 6132.379258] Key type ._llcrypt unregistered [ 6179.441455] Key type ._llcrypt registered [ 6179.449106] Key type .llcrypt registered [ 6179.539649] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_hostid [ 6250.312448] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 6251.612299] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6251.629565] alg: No test for adler32 (adler32-zlib) [ 6252.673892] Lustre: Lustre: Build Version: 2.17.0_RC4_1_g3742a8b [ 6252.986047] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [ 6254.736174] Key type lgssc registered [ 6255.935956] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6274.018323] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6287.322690] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6297.022217] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6307.500902] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6323.111984] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6323.170035] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6323.187349] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6324.432878] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6324.465712] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6324.522592] Lustre: lustre-MDT0000: new disk, initializing [ 6324.618303] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6324.635208] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6328.488210] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 6343.831726] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6343.924492] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6344.044264] Lustre: 245357:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6344.102231] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6344.106363] Lustre: Skipped 1 previous similar message [ 6344.186325] Lustre: lustre-MDT0001: new disk, initializing [ 6344.271058] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6344.311535] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6344.319432] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6347.956953] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 6354.660822] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6358.192238] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6370.802846] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6370.864314] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6371.036993] Lustre: lustre-OST0000: new disk, initializing [ 6371.041795] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6371.118247] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6376.155915] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 6381.102600] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6381.117426] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6381.212391] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6383.623768] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6387.552916] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6387.750932] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6392.105525] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6392.328887] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6396.903569] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6396.905196] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6396.907729] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6396.915697] Lustre: Skipped 1 previous similar message [ 6396.920325] LustreError: Skipped 1 previous similar message [ 6402.023030] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6402.032875] Lustre: Skipped 2 previous similar messages [ 6407.141194] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6407.150322] Lustre: Skipped 1 previous similar message [ 6408.162549] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6408.194397] LustreError: 248987:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6408.291684] Lustre: server umount lustre-OST0000 complete [ 6412.257640] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6412.260109] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6412.269189] Lustre: Skipped 1 previous similar message [ 6412.275680] Lustre: Skipped 1 previous similar message [ 6417.380881] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6417.384924] Lustre: Skipped 1 previous similar message [ 6418.253427] LustreError: 249238:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6418.256295] LustreError: 249238:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6418.308321] Lustre: server umount lustre-MDT0000 complete [ 6422.230745] LustreError: 245349:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767321389 with bad export cookie 12641704251447554712 [ 6422.233147] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6422.239590] LustreError: 245349:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6422.381972] LustreError: 249488:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6422.385140] LustreError: 249488:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6422.674679] Lustre: server umount lustre-MDT0001 complete [ 6434.779698] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6435.288480] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6440.124946] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 6440.418222] LustreError: 250158:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6440.442715] LustreError: 250158:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6445.539797] LustreError: 250158:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6450.659759] LustreError: 250158:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6451.370579] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6451.818419] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6457.438727] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 6463.619239] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6468.374663] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6479.347220] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6479.800983] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6486.622723] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 6492.211904] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6496.142842] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6496.445992] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6500.742026] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6500.989247] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6515.124584] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6515.198815] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6515.343199] Lustre: lustre-OST0001: new disk, initializing [ 6515.347246] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6515.433811] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6522.527367] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 6523.580599] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6523.588242] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6523.696135] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6530.958391] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6531.289575] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6535.848451] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6536.096745] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6540.171914] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8976467db000.ost_server_uuid 50 [ 6541.724666] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8976467db000.ost_server_uuid in IDLE state after 0 sec [ 6549.477315] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6549.484683] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6549.503569] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6549.509681] Lustre: Skipped 1 previous similar message [ 6554.814112] LustreError: 255144:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 6554.817272] LustreError: 255144:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6554.849366] Lustre: server umount lustre-OST0001 complete [ 6558.693643] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6558.701722] LustreError: Skipped 1 previous similar message [ 6558.705077] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6558.716854] Lustre: Skipped 1 previous similar message [ 6559.718860] LustreError: 252143:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6559.743809] LustreError: 252143:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6564.354151] LustreError: 255345:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6564.363683] LustreError: 255345:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6564.591539] Lustre: server umount lustre-OST0000 complete [ 6569.442763] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6569.454727] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6569.477727] Lustre: Skipped 1 previous similar message [ 6569.487599] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6569.497758] Lustre: Skipped 5 previous similar messages [ 6575.426960] LustreError: 255596:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 6575.441551] LustreError: 255596:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6575.659111] Lustre: server umount lustre-MDT0000 complete [ 6579.171298] LustreError: 251064:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6579.195030] LustreError: 251064:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6580.484787] LustreError: 250139:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767321547 with bad export cookie 12641704251447555594 [ 6580.501521] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6580.885171] Lustre: server umount lustre-MDT0001 complete [ 6602.518521] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [ 6604.773562] Key type lgssc unregistered [ 6605.035742] LNet: 257078:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6605.043915] LNetError: 257078:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6606.119877] LNet: Removed LNI 192.168.203.152@tcp [ 6606.982872] Key type .llcrypt unregistered [ 6606.987421] Key type ._llcrypt unregistered [ 6664.297233] Key type ._llcrypt registered [ 6664.304187] Key type .llcrypt registered [ 6664.455839] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_hostid [ 6734.444686] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 6735.680772] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6735.692425] alg: No test for adler32 (adler32-zlib) [ 6736.759740] Lustre: Lustre: Build Version: 2.17.0_RC4_1_g3742a8b [ 6737.037360] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [ 6738.656230] Key type lgssc registered [ 6739.604215] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6757.052410] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6771.197470] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6780.366874] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6789.785591] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6805.852217] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6805.912229] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6805.933710] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6807.252094] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6807.294580] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6807.411306] Lustre: lustre-MDT0000: new disk, initializing [ 6807.493876] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6807.526055] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6812.955391] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 6829.149900] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6829.216679] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6829.290387] Lustre: 267117:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6829.342223] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6829.346545] Lustre: Skipped 1 previous similar message [ 6829.412555] Lustre: lustre-MDT0001: new disk, initializing [ 6829.500657] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6829.535117] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6829.541061] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6833.239975] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 6840.383541] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6843.856271] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6855.596735] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6855.685344] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6855.993748] Lustre: lustre-OST0000: new disk, initializing [ 6856.005396] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6856.117051] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6861.156917] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 6862.923855] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6862.939260] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6863.053643] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6868.110297] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6872.168472] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6872.352269] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6876.712355] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6876.944831] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6878.690737] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6878.693896] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6878.704741] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6883.809643] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6883.818683] Lustre: Skipped 1 previous similar message [ 6884.484117] LustreError: 270747:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6884.590974] Lustre: server umount lustre-OST0000 complete [ 6888.934088] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6888.945404] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6888.949465] Lustre: Skipped 2 previous similar messages [ 6888.974428] Lustre: Skipped 2 previous similar messages [ 6894.054804] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6894.060709] Lustre: Skipped 1 previous similar message [ 6894.650336] LustreError: 270998:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6894.654696] LustreError: 270998:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6894.837221] Lustre: server umount lustre-MDT0000 complete [ 6899.644260] LustreError: 267111:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767321866 with bad export cookie 1933901207163496317 [ 6899.645749] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6899.649348] LustreError: 267111:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6899.768217] LustreError: 271248:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6899.773558] LustreError: 271248:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6899.928653] Lustre: server umount lustre-MDT0001 complete [ 6908.608533] Lustre: DEBUG MARKER: == conf-sanity test 47a: server restart does not lose lru_resize ========================================================== 21:44:34 (1767321874) [ 6939.806176] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_hostid [ 7003.419107] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 7024.984948] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7038.343364] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7048.115439] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7058.031256] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7072.892729] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7073.009929] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7073.262548] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7073.292985] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7073.364805] Lustre: lustre-MDT0000: new disk, initializing [ 7073.429766] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7073.446381] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7077.647511] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 7094.127423] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7094.240917] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7094.391668] Lustre: 279954:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7094.465838] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7094.470805] Lustre: Skipped 1 previous similar message [ 7094.561272] Lustre: lustre-MDT0001: new disk, initializing [ 7094.684335] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7094.716167] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7094.731954] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7098.789843] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 7106.556583] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7110.621814] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7124.608400] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7124.667675] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7124.936936] Lustre: lustre-OST0000: new disk, initializing [ 7124.945863] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7125.047260] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7130.709970] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7130.733964] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7130.877935] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7130.939707] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 7138.023661] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7141.712977] Lustre: Failing over lustre-OST0000 [ 7141.765196] LustreError: 283087:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7141.769407] LustreError: 283087:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 7141.861593] Lustre: server umount lustre-OST0000 complete [ 7160.805683] Lustre: 263584:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767322111/real 1767322111] req@00000000442022ef x1853171123165824/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1767322127 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7160.841380] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7162.336207] Lustre: 263586:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767322113/real 1767322113] req@000000002f996293 x1853171123166080/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1767322129 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7168.486670] Lustre: 263585:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767322118/real 1767322118] req@00000000e3ad270d x1853171123167744/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1767322134 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7168.525557] Lustre: 263585:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 7168.635808] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7169.184811] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7169.211212] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 7170.231372] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 7172.357622] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7172.361548] Lustre: lustre-OST0000: Recovery over after 0:02, of 3 clients 3 recovered and 0 were evicted. [ 7173.408376] Lustre: 263584:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767322123/real 1767322123] req@00000000fe4a522f x1853171123169024/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1767322139 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7173.434883] Lustre: 263584:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 7175.818456] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 7179.514145] Lustre: Failing over lustre-MDT0000 [ 7179.687635] LustreError: 284509:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 7179.700995] LustreError: 284509:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7179.776393] Lustre: server umount lustre-MDT0000 complete [ 7180.775356] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7180.797525] LustreError: 279966:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7180.797536] LustreError: 279966:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7180.832508] Lustre: Skipped 3 previous similar messages [ 7181.683165] LustreError: 279961:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.52@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7181.717551] LustreError: 279961:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7185.897995] LustreError: 279961:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7185.914113] LustreError: 279961:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7191.009488] LustreError: 279966:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7191.026428] LustreError: 279966:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7196.130618] Lustre: 263584:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767322147/real 1767322147] req@000000008a19eb4c x1853171123182208/t0(0) o400->MGC192.168.203.152@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1767322163 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7196.130807] LustreError: 279966:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7196.154362] Lustre: 263584:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 7196.154502] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7196.165067] LustreError: 279966:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7206.369965] LustreError: 279961:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7206.395906] LustreError: 279961:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 7207.685770] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7221.984165] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7222.022026] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 7223.518646] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 7225.611471] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 7227.385854] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7227.391593] Lustre: Skipped 1 previous similar message [ 7227.419337] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 7227.464646] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 7232.491833] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7232.507348] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7232.513329] Lustre: Skipped 1 previous similar message [ 7234.298643] LustreError: 285973:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7234.309018] LustreError: 285973:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7234.480550] Lustre: server umount lustre-OST0000 complete [ 7242.722456] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7242.724595] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7242.749919] Lustre: Skipped 1 previous similar message [ 7242.752971] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7242.772206] Lustre: Skipped 1 previous similar message [ 7245.753579] Lustre: server umount lustre-MDT0000 complete [ 7247.844423] LustreError: 279962:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7247.876689] LustreError: 279962:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 13 previous similar messages [ 7250.786709] LustreError: 279948:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767322217 with bad export cookie 1933901207163498466 [ 7250.793078] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7250.809390] LustreError: 279948:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7250.927550] LustreError: 286473:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7250.931970] LustreError: 286473:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 7251.160974] Lustre: server umount lustre-MDT0001 complete [ 7265.396254] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [ 7269.585136] Key type lgssc unregistered [ 7269.968739] LNet: 287151:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7269.975293] LNetError: 287151:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7270.002509] LNet: Removed LNI 192.168.203.152@tcp [ 7271.110595] Key type .llcrypt unregistered [ 7271.112506] Key type ._llcrypt unregistered [ 7369.670964] Key type ._llcrypt registered [ 7369.673679] Key type .llcrypt registered [ 7369.773842] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 7370.571718] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7370.674124] alg: No test for adler32 (adler32-zlib) [ 7371.734410] Lustre: Lustre: Build Version: 2.17.0_RC4_1_g3742a8b [ 7371.982841] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [ 7373.648191] Key type lgssc registered [ 7374.664778] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7384.272657] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7384.318429] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7385.782350] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7389.886449] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 7395.300031] LustreError: 292518:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7395.308593] LustreError: 292518:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7397.769481] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7400.428806] LustreError: 292518:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7402.207673] LustreError: 292548:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 7402.218288] LustreError: 292548:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 7402.226775] LustreError: 292548:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 17, retries 0, failed: rc = -5 [ 7402.346502] LustreError: 293072:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7402.547941] Lustre: server umount lustre-MDT0000 complete [ 7439.668378] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_hostid [ 7491.104441] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 7510.446065] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7523.836156] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7533.540397] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7543.637116] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7558.988375] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7559.032785] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7559.246554] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7559.278735] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7559.393529] Lustre: lustre-MDT0000: new disk, initializing [ 7559.509360] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7559.522771] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7563.355817] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 7576.672052] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7576.745653] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7576.803223] Lustre: 301431:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7576.848355] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7576.851825] Lustre: Skipped 1 previous similar message [ 7576.900627] Lustre: lustre-MDT0001: new disk, initializing [ 7576.956846] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7577.011383] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7577.031285] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7580.313199] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 7586.361402] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7589.054571] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7600.425293] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7600.484203] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7600.661783] Lustre: lustre-OST0000: new disk, initializing [ 7600.665515] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7600.760427] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7606.021855] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 7609.912103] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7609.921678] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7610.022519] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7612.875644] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7616.041836] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 7617.796885] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 21:56:23 (1767322583) [ 7804.902762] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7804.903141] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7804.911526] Lustre: Skipped 1 previous similar message [ 7804.917491] Lustre: Skipped 2 previous similar messages [ 7809.122990] LustreError: 306217:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 7809.129141] LustreError: 306217:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 7809.283489] Lustre: server umount lustre-MDT0000 complete [ 7810.024143] LustreError: 301443:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7810.042137] LustreError: 301443:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7815.137236] LustreError: 301443:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7815.156467] LustreError: 301443:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7815.759202] LustreError: 303868:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767322782 with bad export cookie 8543335246770799880 [ 7815.759694] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7815.767875] LustreError: 303868:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7820.262241] LustreError: 303116:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7820.271516] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7820.298518] Lustre: Skipped 1 previous similar message [ 7820.304155] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7821.913848] LustreError: 306668:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7821.918184] LustreError: 306668:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7822.041174] Lustre: server umount lustre-MDT0001 complete [ 7837.499561] LustreError: 307264:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 7837.510773] LustreError: 307264:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7837.632942] Lustre: server umount lustre-OST0000 complete [ 7850.239942] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_hostid [ 7903.376860] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 7916.685233] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7923.930267] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7929.408508] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7935.508030] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7946.691183] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7946.749324] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7946.942334] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7946.958709] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7947.016243] Lustre: lustre-MDT0000: new disk, initializing [ 7947.073767] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7947.086241] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7950.227986] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 7963.616166] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7963.676471] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7963.776341] Lustre: 314683:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7963.837178] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7963.842108] Lustre: Skipped 1 previous similar message [ 7963.916365] Lustre: lustre-MDT0001: new disk, initializing [ 7964.023149] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7964.087609] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7964.096801] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7967.792273] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 7973.127190] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7975.958934] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7986.255786] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7986.302501] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7986.460260] Lustre: lustre-OST0000: new disk, initializing [ 7986.466873] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7986.534386] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7990.705768] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 7993.387928] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7993.397110] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7993.492114] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7996.246041] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7999.111733] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7999.233950] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8002.028761] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8002.181559] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8003.554404] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8003.564349] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8003.574143] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8004.064982] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8009.191709] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8009.196855] Lustre: Skipped 2 previous similar messages [ 8009.527478] LustreError: 318312:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 8009.532280] LustreError: 318312:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8009.602556] Lustre: server umount lustre-OST0000 complete [ 8014.307340] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8014.309817] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8014.315517] Lustre: Skipped 2 previous similar messages [ 8014.324442] Lustre: Skipped 2 previous similar messages [ 8024.547071] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8024.552039] Lustre: Skipped 3 previous similar messages [ 8027.104083] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8027.168414] LustreError: 318561:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 8027.174691] LustreError: 318561:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8027.265544] Lustre: server umount lustre-MDT0000 complete [ 8029.667992] LustreError: 316270:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8029.681305] LustreError: 316270:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8030.701145] LustreError: 317119:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767322997 with bad export cookie 8543335246770958710 [ 8030.706129] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8030.709727] LustreError: 317119:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8031.036553] Lustre: server umount lustre-MDT0001 complete [ 8036.006830] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 22:03:22 (1767323002) [ 8061.151293] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_hostid [ 8103.191650] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 8117.074399] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8125.786928] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8132.221586] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8138.586702] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8150.266732] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8150.340866] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8150.631611] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8150.667466] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8150.773465] Lustre: lustre-MDT0000: new disk, initializing [ 8150.868633] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8150.886586] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8154.237713] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 8168.025448] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8168.106845] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8168.226609] Lustre: 327512:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8168.694754] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8173.067351] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 8179.459309] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8182.405281] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8191.966881] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8192.050913] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8192.206888] Lustre: lustre-OST0000: new disk, initializing [ 8192.209544] Lustre: Skipped 1 previous similar message [ 8192.213779] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8192.217739] Lustre: Skipped 2 previous similar messages [ 8196.343313] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 8199.220896] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8199.230909] Lustre: Skipped 1 previous similar message [ 8199.240846] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8199.377719] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8201.944703] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8209.895296] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8209.907562] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8209.908577] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8209.924043] Lustre: Skipped 1 previous similar message [ 8212.734977] LustreError: 330751:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 8212.748506] LustreError: 330751:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 8212.820308] Lustre: server umount lustre-OST0000 complete [ 8220.130584] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8220.131309] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8220.144343] Lustre: Skipped 2 previous similar messages [ 8222.396488] Lustre: server umount lustre-MDT0000 complete [ 8225.249548] LustreError: 328921:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8225.260366] LustreError: 328921:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8225.629925] LustreError: 327506:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767323192 with bad export cookie 8543335246770959592 [ 8225.633156] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8225.639902] LustreError: 327506:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8225.917274] Lustre: server umount lustre-MDT0001 complete [ 8232.260599] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 22:06:38 (1767323198) [ 8257.647133] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_hostid [ 8299.566386] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 8312.269043] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8319.627487] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8326.904806] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8333.649299] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8343.917192] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8343.966949] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8344.120659] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8344.149590] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8344.199875] Lustre: lustre-MDT0000: new disk, initializing [ 8344.252282] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8344.256723] Lustre: Skipped 2 previous similar messages [ 8344.263925] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8346.805691] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 8358.425203] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8358.479463] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8358.544686] Lustre: 339953:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8358.766077] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8361.997380] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 8367.651433] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8370.600864] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8380.064385] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8380.125031] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8384.458236] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 8388.735621] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8389.584266] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8393.697920] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8393.707664] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8393.719426] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8393.722020] Lustre: Skipped 2 previous similar messages [ 8395.646828] LustreError: 343185:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 8395.653230] LustreError: 343185:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 8395.767462] Lustre: server umount lustre-OST0000 complete [ 8413.152138] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8413.272566] Lustre: server umount lustre-MDT0000 complete [ 8414.689342] LustreError: 339964:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8414.699393] LustreError: 339964:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8416.163792] LustreError: 339944:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767323383 with bad export cookie 8543335246770960677 [ 8416.165434] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8416.171640] LustreError: 339944:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8416.442622] Lustre: server umount lustre-MDT0001 complete [ 8425.166631] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [ 8427.024291] Key type lgssc unregistered [ 8427.250712] LNet: 344363:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8427.257287] LNetError: 344363:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8427.267943] LNet: Removed LNI 192.168.203.152@tcp [ 8427.742733] Key type .llcrypt unregistered [ 8427.745318] Key type ._llcrypt unregistered [ 8477.631716] Key type ._llcrypt registered [ 8477.633373] Key type .llcrypt registered [ 8477.698978] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 8478.585343] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8478.643241] alg: No test for adler32 (adler32-zlib) [ 8479.577751] Lustre: Lustre: Build Version: 2.17.0_RC4_1_g3742a8b [ 8479.738686] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [ 8481.360167] Key type lgssc registered [ 8482.753210] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8487.941440] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8487.964636] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8489.263921] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8491.663225] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 8499.086302] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8499.264717] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8501.423112] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 8504.646458] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8506.810827] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8513.093466] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8513.275306] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8515.317851] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 8516.894833] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 8519.966585] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8540.132691] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8540.135327] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8540.138085] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8540.140851] Lustre: Skipped 1 previous similar message [ 8540.154446] Lustre: Skipped 1 previous similar message [ 8545.251636] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8545.254488] Lustre: Skipped 1 previous similar message [ 8545.399084] LustreError: 351468:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8545.462458] Lustre: server umount lustre-OST0000 complete [ 8550.370669] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8550.370940] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8550.378986] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8550.387068] Lustre: Skipped 1 previous similar message [ 8550.395599] Lustre: Skipped 1 previous similar message [ 8555.493975] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8555.498318] Lustre: Skipped 1 previous similar message [ 8560.610542] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8560.618652] Lustre: Skipped 1 previous similar message [ 8562.144197] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8562.199482] LustreError: 351719:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 8562.207729] LustreError: 351719:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8562.244044] Lustre: server umount lustre-MDT0000 complete [ 8565.109232] LustreError: 348795:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767323532 with bad export cookie 17750847960586448324 [ 8565.111989] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8565.124827] LustreError: 348795:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8565.219358] LustreError: 351970:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 8565.223340] LustreError: 351970:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8565.419420] Lustre: server umount lustre-MDT0001 complete [ 8573.408324] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [ 8574.887891] Key type lgssc unregistered [ 8575.092832] LNet: 352646:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8575.098932] LNetError: 352646:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8575.114933] LNet: Removed LNI 192.168.203.152@tcp [ 8575.567855] Key type .llcrypt unregistered [ 8575.569250] Key type ._llcrypt unregistered [ 8621.431340] Key type ._llcrypt registered [ 8621.432702] Key type .llcrypt registered [ 8621.501903] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 8621.895846] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8621.957850] alg: No test for adler32 (adler32-zlib) [ 8622.896462] Lustre: Lustre: Build Version: 2.17.0_RC4_1_g3742a8b [ 8623.038859] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [ 8624.656317] Key type lgssc registered [ 8625.246568] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8629.125111] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8629.136677] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8630.407584] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8632.381667] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 8639.246201] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8639.453652] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8641.624318] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 8644.962647] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8646.906307] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8653.145346] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8653.360418] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8655.413292] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 8656.497388] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 8659.581756] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8669.667746] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8669.668032] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8669.669374] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8669.671391] LustreError: Skipped 1 previous similar message [ 8669.682355] Lustre: Skipped 1 previous similar message [ 8672.100591] Lustre: lustre-OST0000: Not available for connect from 192.168.203.52@tcp (stopping) [ 8672.105988] Lustre: Skipped 1 previous similar message [ 8674.159915] LustreError: 359726:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8674.207685] Lustre: server umount lustre-OST0000 complete [ 8676.915915] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff8976464fe000.ost_server_uuid 50 [ 8677.791168] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8976464fe000.ost_server_uuid in DISCONN state after 0 sec [ 8678.756573] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 8690.661814] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8690.669466] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8690.682391] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8694.242904] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8694.245143] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8695.944957] LustreError: 360169:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 8695.948079] LustreError: 360169:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8698.019634] LustreError: 360169:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 8698.029140] LustreError: 360169:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8698.072811] Lustre: server umount lustre-MDT0000 complete [ 8699.361070] LustreError: 357097:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8699.370871] LustreError: 357097:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 8700.935924] LustreError: 357077:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767323668 with bad export cookie 3618850477122777838 [ 8700.937179] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8700.943126] LustreError: 357077:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8700.953099] LustreError: 358003:0:(osp_precreate.c:651:osp_precreate_send()) lustre-OST0000-osc-MDT0001: can't precreate: rc = -5 [ 8700.962509] LustreError: 358003:0:(osp_precreate.c:1355:osp_precreate_thread()) lustre-OST0000-osc-MDT0001: cannot precreate objects: rc = -5 [ 8701.035835] LustreError: 360421:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 8701.039193] LustreError: 360421:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 8701.135156] Lustre: server umount lustre-MDT0001 complete [ 8705.717721] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 22:14:32 (1767323672) [ 8711.088294] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8711.310361] LustreError: 361284:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8711.359858] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8713.216298] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 8716.769549] LustreError: 361284:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8719.600353] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8719.815256] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8721.644759] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 8724.048713] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8725.658324] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8730.674166] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8730.860379] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8733.730210] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 8736.478777] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8737.005796] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8743.842274] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8743.893093] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8743.981301] Lustre: lustre-OST0001: new disk, initializing [ 8743.984472] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8746.970377] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 8751.008535] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 8751.640095] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8751.645844] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8751.683496] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8752.611163] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8752.616708] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8752.624449] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8752.629219] Lustre: Skipped 1 previous similar message [ 8755.681503] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8758.323500] LustreError: 365215:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8758.328085] LustreError: 365215:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 8758.356975] Lustre: server umount lustre-OST0000 complete [ 8760.930568] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8761.031815] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 8761.679957] LustreError: 363263:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.52@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8763.015848] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8763.118442] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 8766.799941] LustreError: 365146:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.52@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8766.809380] LustreError: 365146:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8777.187241] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8777.187643] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8777.187915] LustreError: 363263:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8777.187926] LustreError: 363263:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 8777.190144] LustreError: Skipped 1 previous similar message [ 8777.195833] Lustre: Skipped 1 previous similar message [ 8777.197324] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8777.210682] Lustre: Skipped 5 previous similar messages [ 8779.565113] LustreError: 365918:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 8779.568146] LustreError: 365918:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8779.625510] Lustre: server umount lustre-OST0001 complete [ 8796.128213] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8796.176512] LustreError: 366168:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 8796.179405] LustreError: 366168:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8796.215096] Lustre: server umount lustre-MDT0000 complete [ 8797.670323] LustreError: 362188:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8797.682249] LustreError: 362188:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8798.371264] LustreError: 363285:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767323765 with bad export cookie 3618850477122778895 [ 8798.373772] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8798.375034] LustreError: 363285:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8798.541760] Lustre: server umount lustre-MDT0001 complete [ 8801.944540] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8804.122372] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8806.649978] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8809.115242] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8812.113878] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 22:16:18 (1767323778) [ 8817.049279] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8817.222547] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8817.241957] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8817.341720] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8817.344599] Lustre: Skipped 1 previous similar message [ 8819.353741] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 8825.334837] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8825.386630] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8825.405875] Lustre: 368296:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8827.357365] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 8830.023403] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8831.733894] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8836.229953] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8836.327584] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8837.880097] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8839.003187] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 8841.912113] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8846.064143] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8846.095758] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 8846.149988] LustreError: 370312:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8848.674396] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 8851.213703] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 8853.473520] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8853.473736] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8853.477386] LustreError: Skipped 1 previous similar message [ 8853.482877] Lustre: Skipped 3 previous similar messages [ 8853.486376] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8853.489908] Lustre: Skipped 7 previous similar messages [ 8858.481277] LustreError: 372061:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 8858.483355] LustreError: 372061:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 8858.506510] Lustre: server umount lustre-OST0000 complete [ 8871.393080] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8871.395995] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8871.401149] Lustre: Skipped 1 previous similar message [ 8875.991616] Lustre: server umount lustre-OST0001 complete [ 8884.539170] Lustre: server umount lustre-MDT0000 complete [ 8886.397233] LustreError: 369221:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767323853 with bad export cookie 3618850477122780281 [ 8886.400578] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8886.404556] LustreError: 369221:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8886.532617] Lustre: server umount lustre-MDT0001 complete [ 8889.241811] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8891.120191] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8892.977214] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8894.943341] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8897.963567] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 22:17:44 (1767323864) [ 8918.307444] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_hostid [ 8949.032959] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 8958.439798] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8964.450734] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8968.848691] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8973.467311] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8980.552267] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8980.582817] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8980.701515] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8980.720069] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8980.761476] Lustre: lustre-MDT0000: new disk, initializing [ 8980.806547] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8980.813598] Lustre: Skipped 3 previous similar messages [ 8980.829761] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8982.666813] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 8990.045237] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8990.103053] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8990.147783] Lustre: 382687:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8990.173554] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8990.176396] Lustre: Skipped 1 previous similar message [ 8990.213492] Lustre: lustre-MDT0001: new disk, initializing [ 8990.276689] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8990.281607] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8991.989699] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 8995.728719] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8997.502130] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9003.425356] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9003.450263] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9003.558469] Lustre: lustre-OST0000: new disk, initializing [ 9003.561254] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9006.196178] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 9010.198790] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9010.202933] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9010.252111] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9010.262817] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9012.419939] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9012.514165] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9014.203703] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9014.297272] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9015.269853] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9015.270611] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9015.274530] LustreError: Skipped 1 previous similar message [ 9015.281602] Lustre: Skipped 2 previous similar messages [ 9015.283422] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9015.287384] Lustre: Skipped 8 previous similar messages [ 9017.138085] LustreError: 386315:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 9017.140961] LustreError: 386315:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 9017.202981] Lustre: server umount lustre-OST0000 complete [ 9025.386592] Lustre: server umount lustre-MDT0000 complete [ 9025.505029] LustreError: 382699:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9025.516466] LustreError: 382699:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 9027.905684] LustreError: 383814:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767323995 with bad export cookie 3618850477122781814 [ 9027.908640] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9027.913086] LustreError: 383814:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9028.117878] Lustre: server umount lustre-MDT0001 complete [ 9034.207886] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9036.389151] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 9042.416045] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9044.171614] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 9046.531828] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9048.286192] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9053.652680] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9053.882380] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9053.884896] Lustre: Skipped 4 previous similar messages [ 9056.624827] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 9059.111152] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9060.934397] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9061.017761] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9062.927907] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9068.199896] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 5 sec [ 9075.203116] Lustre: server umount lustre-OST0000 complete [ 9077.914475] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9079.558166] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9106.515403] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9109.099914] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 9111.257969] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9112.802225] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 9112.808199] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9122.021753] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 9122.033344] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 9125.727476] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 13 sec [ 9127.394102] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9127.402161] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9127.405155] Lustre: Skipped 5 previous similar messages [ 9127.409716] Lustre: Skipped 5 previous similar messages [ 9127.418235] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9127.421049] LustreError: Skipped 2 previous similar messages [ 9133.209612] Lustre: server umount lustre-OST0000 complete [ 9141.597980] Lustre: server umount lustre-MDT0000 complete [ 9143.675500] LustreError: 391799:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767324110 with bad export cookie 3618850477122782696 [ 9143.678079] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9143.680495] LustreError: 391799:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9143.822085] Lustre: server umount lustre-MDT0001 complete [ 9147.233542] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 22:21:54 (1767324114) [ 9151.407534] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9153.517107] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 9157.092697] LustreError: 393857:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9157.102818] LustreError: 393857:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 9159.596334] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9161.419370] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 9163.918622] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9165.444606] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9170.001017] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9172.718658] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 9175.038413] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9176.654421] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9176.738685] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9178.419261] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9185.714283] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 7 sec [ 9191.082026] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9191.118846] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9191.183864] Lustre: lustre-OST0001: new disk, initializing [ 9191.186766] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9191.220610] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9191.223075] Lustre: Skipped 4 previous similar messages [ 9193.669724] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 9197.414540] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 9199.122784] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 9199.633950] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9199.638671] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9199.688516] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9200.287971] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 9202.151811] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 9202.250750] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9205.363158] LustreError: 398841:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 9205.366222] LustreError: 398841:0:(obd_class.h:479:obd_check_dev()) Skipped 39 previous similar messages [ 9205.392563] Lustre: server umount lustre-OST0001 complete [ 9208.195184] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 9208.302958] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 9210.117424] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 9210.217847] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 9236.405652] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9239.072541] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 9241.515967] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 9241.570869] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 9241.583874] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 9241.585054] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9241.600077] Lustre: Skipped 1 previous similar message [ 9242.403476] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 9249.470645] Lustre: server umount lustre-OST0001 complete [ 9262.561172] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9262.564932] LustreError: Skipped 4 previous similar messages [ 9262.567052] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9262.572780] Lustre: Skipped 6 previous similar messages [ 9262.575591] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9262.578052] Lustre: Skipped 10 previous similar messages [ 9276.384331] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9276.452759] Lustre: server umount lustre-OST0000 complete [ 9284.525718] Lustre: server umount lustre-MDT0000 complete [ 9286.786328] LustreError: 393837:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767324253 with bad export cookie 3618850477122783662 [ 9286.787089] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9286.792167] LustreError: 393837:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9293.020214] Lustre: server umount lustre-MDT0001 complete [ 9296.064707] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9298.500671] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9300.603277] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9302.818401] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9306.068181] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 22:24:32 (1767324272) [ 9310.276605] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9310.409572] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9310.426428] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9312.035199] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 9316.918078] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9316.964954] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9316.984269] Lustre: 403290:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9318.786181] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 9321.019390] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9322.631976] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9327.295105] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9327.380759] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9329.711389] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 9331.838761] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9343.483469] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9343.530237] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 9346.033587] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 9348.542941] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 9350.167164] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 1 sec [ 9351.680914] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 9353.811651] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 9355.463497] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 9355.542750] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9357.037922] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8976489d7800.ost_server_uuid 50 [ 9357.765850] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8976489d7800.ost_server_uuid in FULL state after 0 sec [ 9358.246382] Lustre: Permanently deactivating lustre-OST0001 [ 9358.254549] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 9359.271232] Lustre: Permanently reactivating lustre-OST0001 [ 9366.529843] Lustre: server umount lustre-OST0001 complete [ 9382.368110] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9382.439518] Lustre: server umount lustre-OST0000 complete [ 9390.763301] Lustre: server umount lustre-MDT0000 complete [ 9393.208021] LustreError: 403280:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767324360 with bad export cookie 3618850477122784950 [ 9393.210951] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9393.213669] LustreError: 403280:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9393.409375] Lustre: server umount lustre-MDT0001 complete [ 9396.490348] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9398.607409] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9400.754085] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9402.860502] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9405.850121] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 22:26:12 (1767324372) [ 9406.684967] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9411.758695] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9411.945130] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9411.970109] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9411.975526] Lustre: Skipped 5 previous similar messages [ 9414.086635] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 9419.566932] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9419.630216] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9419.659638] Lustre: 410551:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9421.739730] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 9424.351367] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9425.994045] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9431.555613] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9431.598623] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9431.750594] Lustre: Permanently deactivating lustre-OST0000 [ 9431.767837] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 9435.007233] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 9437.712520] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9440.762872] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 9442.511491] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9445.171500] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 9447.953994] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 9449.019101] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 9453.964375] Lustre: Permanently reactivating lustre-OST0000 [ 9453.972595] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 9453.975034] Lustre: Skipped 2 previous similar messages [ 9454.435442] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 9454.438576] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 9454.444413] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9454.463078] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 9454.471044] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 9464.326542] LustreError: 414392:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 9464.333981] LustreError: 414392:0:(obd_class.h:479:obd_check_dev()) Skipped 55 previous similar messages [ 9464.360520] Lustre: server umount lustre-OST0001 complete [ 9469.411930] LustreError: 412561:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9469.418320] LustreError: 412561:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 35 previous similar messages [ 9472.287249] Lustre: server umount lustre-OST0000 complete [ 9480.484690] Lustre: server umount lustre-MDT0000 complete [ 9482.500407] LustreError: 412584:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767324449 with bad export cookie 3618850477122786616 [ 9482.504194] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9482.506111] LustreError: 412584:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9482.662708] Lustre: server umount lustre-MDT0001 complete [ 9488.025636] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [ 9489.223136] Key type lgssc unregistered [ 9489.374499] LNet: 415769:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9489.377425] LNetError: 415769:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9489.389306] LNet: Removed LNI 192.168.203.152@tcp [ 9489.716448] Key type .llcrypt unregistered [ 9489.717974] Key type ._llcrypt unregistered [ 9520.621204] Key type ._llcrypt registered [ 9520.623028] Key type .llcrypt registered [ 9520.666672] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 9521.026523] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9521.080026] alg: No test for adler32 (adler32-zlib) [ 9521.959794] Lustre: Lustre: Build Version: 2.17.0_RC4_1_g3742a8b [ 9522.058206] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [ 9523.664288] Key type lgssc registered [ 9524.140444] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9525.884802] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9530.549221] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9530.560781] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9531.776544] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9533.414639] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 9538.341978] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9538.369855] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9538.410382] Lustre: Found index 1 for lustre-MDT0001, updating log [ 9538.423282] Lustre: 420309:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9538.430385] Lustre: Permanently deactivating lustre-MDT0001 [ 9538.439108] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 9538.564692] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9540.086758] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 9540.464024] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 9540.468889] LustreError: 420352:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 9540.473977] LustreError: 420352:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 9540.478353] LustreError: 420352:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 9, retries 1, failed: rc = -108 [ 9542.192754] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9543.627212] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9547.627029] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9547.769734] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9548.772291] LustreError: 422325:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9548.775989] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 9548.779210] LustreError: 422325:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9550.214289] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 9552.468837] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9553.891802] LustreError: 422461:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9555.939325] LustreError: 422461:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9555.945748] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 9555.951136] LustreError: 422461:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9557.086361] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9557.208546] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9559.624440] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 9561.853458] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 9562.602896] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 9562.603723] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 9564.668753] Lustre: Permanently reactivating lustre-MDT0001 [ 9564.676323] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 9564.678988] Lustre: Skipped 2 previous similar messages [ 9568.669321] Lustre: lustre-MDT0001: Client f358cd05-b552-41ae-a045-c7b5c59eca04 (at 192.168.203.52@tcp) reconnecting [ 9570.316886] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9571.115173] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9573.469133] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 9573.578858] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 9575.523067] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 9575.619281] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 9576.697893] Lustre: Permanently deactivating lustre-MDT0001 [ 9576.706608] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 9576.710452] Lustre: Skipped 2 previous similar messages [ 9583.643399] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 9588.195370] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9588.200087] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9588.201805] Lustre: Skipped 2 previous similar messages [ 9588.204161] Lustre: Skipped 1 previous similar message [ 9592.650100] LustreError: 425016:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 9592.681985] Lustre: server umount lustre-MDT0000 complete [ 9593.314513] LustreError: 420318:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9593.329671] LustreError: 420318:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9594.803131] LustreError: 420302:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767324561 with bad export cookie 6179988191580316241 [ 9594.804993] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9594.807860] LustreError: 420302:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9594.886831] LustreError: 425266:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 9594.889330] LustreError: 425266:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 9594.950246] Lustre: server umount lustre-MDT0001 complete [ 9602.417150] LustreError: 425466:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 9602.420616] LustreError: 425466:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 9602.438434] Lustre: server umount lustre-OST0000 complete [ 9607.473627] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 22:29:34 (1767324574) [ 9619.544987] LustreError: 427362:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 9619.547648] LustreError: 427362:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 9619.601199] Lustre: server umount lustre-OST0001 complete [ 9624.866387] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_hostid [ 9655.588997] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 9664.268762] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9668.526246] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9672.356381] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9676.279351] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9682.567048] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9682.592111] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9682.684287] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9682.687813] Lustre: Skipped 2 previous similar messages [ 9682.698045] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9682.734771] Lustre: lustre-MDT0000: new disk, initializing [ 9682.764117] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9682.772705] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9684.197536] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 9691.812327] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9691.845568] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9691.917662] Lustre: 434560:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9691.952638] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9691.954913] Lustre: Skipped 1 previous similar message [ 9691.992816] Lustre: lustre-MDT0001: new disk, initializing [ 9692.037028] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9692.059249] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9692.068100] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9693.769598] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 9697.539272] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9699.056413] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9704.880860] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9704.914204] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9705.046796] Lustre: lustre-OST0000: new disk, initializing [ 9705.049608] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9707.376994] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 9710.964435] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9714.175261] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9714.179776] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9714.216179] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9717.135433] LustreError: 434566:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 9725.522168] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9725.564729] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9725.641706] Lustre: lustre-OST0001: new disk, initializing [ 9725.644060] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9725.702843] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9725.705416] Lustre: Skipped 1 previous similar message [ 9727.136154] LustreError: 434566:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 9727.144701] LustreError: 434566:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 9728.188480] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 9732.111512] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9732.117679] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9732.155370] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9732.243274] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 9733.010514] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 9737.145268] LustreError: 434566:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 9738.072559] Lustre: lustre-OST0001: Not available for connect from 192.168.203.52@tcp (stopping) [ 9738.075364] Lustre: Skipped 2 previous similar messages [ 9742.308641] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9742.308780] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9742.312927] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9742.320298] Lustre: Skipped 2 previous similar messages [ 9744.045635] LustreError: 439089:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 9744.048098] LustreError: 439089:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 9744.067854] Lustre: server umount lustre-OST0001 complete [ 9747.428479] LustreError: 436983:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9747.434373] LustreError: 436983:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9757.667643] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9757.668196] LustreError: 437606:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9757.669627] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9757.669633] Lustre: Skipped 2 previous similar messages [ 9757.672278] Lustre: Skipped 1 previous similar message [ 9757.678184] LustreError: 437606:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9762.368437] LustreError: 439312:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 9762.371309] LustreError: 439312:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 9762.431294] Lustre: server umount lustre-OST0000 complete [ 9767.906062] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9767.906669] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9767.912149] Lustre: Skipped 1 previous similar message [ 9767.917299] Lustre: Skipped 2 previous similar messages [ 9770.673415] Lustre: server umount lustre-MDT0000 complete [ 9772.667223] LustreError: 434551:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767324739 with bad export cookie 6179988191580319048 [ 9772.671102] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9772.671677] LustreError: 434551:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9778.820682] LustreError: 439813:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 9778.823341] LustreError: 439813:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 9778.891447] Lustre: server umount lustre-MDT0001 complete [ 9783.861869] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [ 9785.001527] Key type lgssc unregistered [ 9785.129386] LNet: 440487:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9785.133343] LNetError: 440487:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9785.146318] LNet: Removed LNI 192.168.203.152@tcp [ 9785.470633] Key type .llcrypt unregistered [ 9785.471846] Key type ._llcrypt unregistered [ 9807.624682] Key type ._llcrypt registered [ 9807.626197] Key type .llcrypt registered [ 9807.663608] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_hostid [ 9836.869979] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 9837.218811] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9837.251383] alg: No test for adler32 (adler32-zlib) [ 9838.113378] Lustre: Lustre: Build Version: 2.17.0_RC4_1_g3742a8b [ 9838.199642] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [ 9839.784132] Key type lgssc registered [ 9840.178753] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9847.287403] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9852.045074] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9855.967229] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9859.737069] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9865.775517] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9865.797473] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9865.802690] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9866.896195] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9866.910594] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9866.948881] Lustre: lustre-MDT0000: new disk, initializing [ 9866.978857] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9866.988407] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9868.456153] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 9875.307381] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9875.338022] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9875.376980] Lustre: 450509:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9875.401880] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9875.404446] Lustre: Skipped 1 previous similar message [ 9875.441801] Lustre: lustre-MDT0001: new disk, initializing [ 9875.476098] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9875.492078] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9875.495696] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9876.828227] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 9880.050701] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9881.292192] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9886.079236] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9886.106665] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9886.213283] Lustre: lustre-OST0000: new disk, initializing [ 9886.216095] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9886.255960] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9887.748719] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9887.752787] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9887.785196] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9888.236563] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 9891.665554] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9893.144739] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9893.225729] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9894.644068] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9894.726783] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9897.953248] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9897.953672] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9897.956489] LustreError: Skipped 1 previous similar message [ 9897.962498] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9897.966267] Lustre: Skipped 1 previous similar message [ 9901.555881] LustreError: 454138:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 9901.593804] Lustre: server umount lustre-OST0000 complete [ 9906.145200] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9906.148831] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9906.152984] Lustre: Skipped 1 previous similar message [ 9906.156365] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9908.193471] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9908.193622] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9908.204488] Lustre: Skipped 1 previous similar message [ 9909.311593] LustreError: 454387:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 9909.314035] LustreError: 454387:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 9909.344329] Lustre: server umount lustre-MDT0000 complete [ 9911.102195] LustreError: 452946:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767324878 with bad export cookie 7627199697844157431 [ 9911.104899] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9911.106838] LustreError: 452946:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9911.169713] LustreError: 454636:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 9911.172368] LustreError: 454636:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 9911.227975] Lustre: server umount lustre-MDT0001 complete [ 9914.270346] Lustre: DEBUG MARKER: == conf-sanity test 52: check recovering objects from lost+found ========================================================== 22:34:41 (1767324881) [ 9918.178082] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9918.377536] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9919.735589] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 9923.553237] LustreError: 455498:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9923.560552] LustreError: 455498:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9924.084593] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9924.211376] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9925.750169] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 9927.710607] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9929.063562] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9932.746655] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9932.868447] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9934.955584] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 9936.892413] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9987.554743] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9987.555833] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9987.558799] Lustre: Skipped 1 previous similar message [ 9992.111116] LustreError: 458227:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 9992.113781] LustreError: 458227:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 9992.162849] Lustre: server umount lustre-OST0000 complete [ 9993.751023] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 9999.146817] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9999.271554] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10001.297715] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [10003.207421] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10007.461322] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [10007.464660] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [10007.464674] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [10007.470386] Lustre: Skipped 1 previous similar message [10036.195947] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10036.201210] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10036.203594] Lustre: Skipped 2 previous similar messages [10042.159053] LustreError: 460082:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [10042.161230] LustreError: 460082:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [10042.201247] Lustre: server umount lustre-OST0000 complete [10046.433843] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10046.438506] Lustre: Skipped 1 previous similar message [10046.440535] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10046.442262] Lustre: Skipped 3 previous similar messages [10049.753439] Lustre: server umount lustre-MDT0000 complete [10051.320295] LustreError: 455479:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767325018 with bad export cookie 7627199697844158306 [10051.323674] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10051.324302] LustreError: 455479:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [10051.401247] LustreError: 460580:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [10051.403483] LustreError: 460580:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [10051.461759] Lustre: server umount lustre-MDT0001 complete [10056.052780] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [10057.118340] Key type lgssc unregistered [10057.240396] LNet: 461256:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10057.242466] LNetError: 461256:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [10057.249331] LNet: Removed LNI 192.168.203.152@tcp [10057.535636] Key type .llcrypt unregistered [10057.537214] Key type ._llcrypt unregistered [10084.778960] Key type ._llcrypt registered [10084.780133] Key type .llcrypt registered [10084.814613] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [10085.238420] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10085.245025] alg: No test for adler32 (adler32-zlib) [10086.100659] Lustre: Lustre: Build Version: 2.17.0_RC4_1_g3742a8b [10086.183096] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [10087.760197] Key type lgssc registered [10088.093748] Lustre: Echo OBD driver; http://www.lustre.org/ [10091.774784] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10091.782454] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10092.944504] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10094.308744] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [10098.462065] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10098.578679] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10099.857105] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [10101.565865] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10102.750406] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10106.202601] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10106.329219] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10108.265312] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [10110.263031] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10118.630881] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [10128.865735] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10128.867477] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10128.868648] LustreError: Skipped 1 previous similar message [10128.868820] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10128.874194] Lustre: Skipped 1 previous similar message [10133.986228] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10133.988899] Lustre: Skipped 2 previous similar messages [10134.830824] LustreError: 469128:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [10134.868325] Lustre: server umount lustre-OST0000 complete [10139.105616] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10139.111545] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10139.113383] Lustre: Skipped 1 previous similar message [10142.655587] LustreError: 469378:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [10142.657694] LustreError: 469378:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [10142.690307] Lustre: server umount lustre-MDT0000 complete [10144.224949] LustreError: 465705:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10144.232559] LustreError: 465705:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10144.340167] LustreError: 467708:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767325111 with bad export cookie 10580171637714412496 [10144.343560] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10144.343987] LustreError: 467708:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [10144.411375] LustreError: 469629:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [10144.412882] LustreError: 469629:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [10144.468864] Lustre: server umount lustre-MDT0001 complete [10149.005282] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [10149.961059] Key type lgssc unregistered [10150.086424] LNet: 470304:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10150.089860] LNetError: 470304:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [10150.103503] LNet: Removed LNI 192.168.203.152@tcp [10150.417952] Key type .llcrypt unregistered [10150.419297] Key type ._llcrypt unregistered [10177.285249] Key type ._llcrypt registered [10177.286656] Key type .llcrypt registered [10177.326719] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [10177.646306] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10177.689914] alg: No test for adler32 (adler32-zlib) [10178.537071] Lustre: Lustre: Build Version: 2.17.0_RC4_1_g3742a8b [10178.612833] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [10180.192112] Key type lgssc registered [10180.556062] Lustre: Echo OBD driver; http://www.lustre.org/ [10183.062484] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10183.068825] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10184.213900] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10185.496157] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [10189.356109] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10189.475850] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10190.873661] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [10192.751070] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10193.997433] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10197.458442] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10197.565856] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10199.408500] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [10201.108581] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10209.895690] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [10215.394180] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10215.395271] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10215.396986] LustreError: Skipped 1 previous similar message [10215.401783] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10215.404821] Lustre: Skipped 1 previous similar message [10220.515287] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10220.517566] Lustre: Skipped 1 previous similar message [10221.229087] LustreError: 477352:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [10221.262718] Lustre: server umount lustre-OST0000 complete [10225.633558] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10225.633650] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10225.634506] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10225.642069] Lustre: Skipped 2 previous similar messages [10228.925414] LustreError: 477601:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [10228.927682] LustreError: 477601:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [10228.953667] Lustre: server umount lustre-MDT0000 complete [10230.477247] LustreError: 474539:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767325197 with bad export cookie 16800244320369318239 [10230.479713] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10230.481343] LustreError: 474539:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [10230.546068] LustreError: 477851:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [10230.548669] LustreError: 477851:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [10230.599978] Lustre: server umount lustre-MDT0001 complete [10235.053525] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [10236.072744] Key type lgssc unregistered [10236.196429] LNet: 478527:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10236.198722] LNetError: 478527:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [10236.205428] LNet: Removed LNI 192.168.203.152@tcp [10236.514505] Key type .llcrypt unregistered [10236.515803] Key type ._llcrypt unregistered [10263.511894] Key type ._llcrypt registered [10263.513352] Key type .llcrypt registered [10263.548698] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [10263.992930] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10264.003530] alg: No test for adler32 (adler32-zlib) [10264.854898] Lustre: Lustre: Build Version: 2.17.0_RC4_1_g3742a8b [10264.933740] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [10266.512139] Key type lgssc registered [10266.880404] Lustre: Echo OBD driver; http://www.lustre.org/ [10269.434429] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10269.441384] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10270.599167] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10271.975237] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [10276.128686] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10276.261157] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10277.592658] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [10279.448325] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10280.673173] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10284.158272] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10284.264485] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10286.117658] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [10287.786251] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10295.525923] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [10306.020229] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10306.021026] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10306.021669] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10306.024579] Lustre: Skipped 1 previous similar message [10307.503084] LustreError: 486398:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [10307.540114] Lustre: server umount lustre-OST0000 complete [10311.137660] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10311.138685] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10311.142125] Lustre: Skipped 1 previous similar message [10311.145686] Lustre: Skipped 2 previous similar messages [10315.197662] LustreError: 486647:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [10315.199300] LustreError: 486647:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [10315.223968] Lustre: server umount lustre-MDT0000 complete [10316.256961] LustreError: 484376:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10316.261966] LustreError: 484376:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10316.760247] LustreError: 486650:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767325283 with bad export cookie 18144907854884712760 [10316.763020] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10316.764185] LustreError: 486650:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [10316.836503] LustreError: 486899:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [10316.839557] LustreError: 486899:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [10316.897343] Lustre: server umount lustre-MDT0001 complete [10321.289874] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [10322.231483] Key type lgssc unregistered [10322.351378] LNet: 487575:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10322.354552] LNetError: 487575:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [10322.363308] LNet: Removed LNI 192.168.203.152@tcp [10322.626031] Key type .llcrypt unregistered [10322.627256] Key type ._llcrypt unregistered [10347.736997] Key type ._llcrypt registered [10347.738115] Key type .llcrypt registered [10347.782542] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [10348.089891] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10348.119025] alg: No test for adler32 (adler32-zlib) [10348.967367] Lustre: Lustre: Build Version: 2.17.0_RC4_1_g3742a8b [10349.048261] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [10350.624168] Key type lgssc registered [10350.998157] Lustre: Echo OBD driver; http://www.lustre.org/ [10353.722157] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10353.728818] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10354.921179] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10356.239776] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [10360.341447] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10360.474605] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10361.778893] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [10363.514398] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10364.678767] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10368.053019] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10368.159571] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10369.986601] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [10371.759568] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10380.454642] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [10385.799138] LustreError: 494659:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [10385.836713] Lustre: server umount lustre-OST0000 complete [10391.010613] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10391.011986] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10391.015259] Lustre: Skipped 1 previous similar message [10396.129537] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10396.131801] Lustre: Skipped 2 previous similar messages [10397.156011] LustreError: 494908:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [10397.158048] LustreError: 494908:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [10397.181987] Lustre: server umount lustre-MDT0000 complete [10399.682146] LustreError: 494910:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767325366 with bad export cookie 5738203341981834031 [10399.684856] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10399.686676] LustreError: 494910:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [10408.932971] LustreError: 495161:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [10408.935207] LustreError: 495161:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [10408.993533] Lustre: server umount lustre-MDT0001 complete [10413.611968] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [10414.702810] Key type lgssc unregistered [10414.836331] LNet: 495837:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10414.838687] LNetError: 495837:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [10414.847434] LNet: Removed LNI 192.168.203.152@tcp [10415.158294] Key type .llcrypt unregistered [10415.159683] Key type ._llcrypt unregistered [10419.215695] Key type ._llcrypt registered [10419.217194] Key type .llcrypt registered [10419.255078] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [10432.338650] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_hostid [10456.712620] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [10457.139080] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10457.147578] alg: No test for adler32 (adler32-zlib) [10458.017203] Lustre: Lustre: Build Version: 2.17.0_RC4_1_g3742a8b [10458.101618] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [10459.688132] Key type lgssc registered [10460.069757] Lustre: Echo OBD driver; http://www.lustre.org/ [10466.504196] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10470.529890] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10473.793620] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10477.043483] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10482.214161] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10482.233353] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10482.240212] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10483.332446] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10483.345813] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10483.385499] Lustre: lustre-MDT0000: new disk, initializing [10483.412794] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10483.420614] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10484.713273] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [10491.099865] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10491.125580] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10491.162397] Lustre: 505073:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10491.186812] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10491.188776] Lustre: Skipped 1 previous similar message [10491.231893] Lustre: lustre-MDT0001: new disk, initializing [10491.271874] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10491.286146] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10491.289397] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10492.523174] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [10495.515135] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10496.691543] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10501.063314] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10501.086040] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10501.185824] Lustre: lustre-OST0000: new disk, initializing [10501.187604] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10501.221428] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10503.157820] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [10506.294752] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10507.265807] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10507.270558] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10507.307153] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10507.671803] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10507.753598] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10509.091444] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10509.173206] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10512.355371] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10512.356420] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10512.358375] LustreError: Skipped 1 previous similar message [10512.363505] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10512.366538] Lustre: Skipped 1 previous similar message [10515.888110] LustreError: 508701:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [10515.931870] Lustre: server umount lustre-OST0000 complete [10522.080872] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10522.082994] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10522.086495] Lustre: Skipped 1 previous similar message [10522.088597] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10523.776635] LustreError: 508951:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [10523.778759] LustreError: 508951:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [10523.810582] Lustre: server umount lustre-MDT0000 complete [10525.502263] LustreError: 505065:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767325492 with bad export cookie 11431637541420308781 [10525.505362] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10525.506826] LustreError: 505065:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [10525.580528] LustreError: 509201:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [10525.582445] LustreError: 509201:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [10525.640269] Lustre: server umount lustre-MDT0001 complete [10528.468940] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 22:44:55 (1767325495) [10532.006135] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10532.210847] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10533.582243] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [10537.441097] LustreError: 510062:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10537.446864] LustreError: 510062:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10537.826355] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10537.960533] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10539.279155] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [10541.171296] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10542.374878] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10546.068057] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10546.178170] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10548.102490] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [10549.834908] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10563.553025] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10563.555127] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10563.558158] Lustre: Skipped 1 previous similar message [10563.559525] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10563.560928] Lustre: Skipped 1 previous similar message [10567.650205] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10567.651564] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10567.652172] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10567.658536] Lustre: Skipped 1 previous similar message [10569.069055] LustreError: 512718:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [10569.070681] LustreError: 512718:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [10569.100795] Lustre: server umount lustre-OST0000 complete [10572.769567] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10572.773596] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10572.775420] Lustre: Skipped 1 previous similar message [10576.702935] LustreError: 512967:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [10576.705729] LustreError: 512967:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [10576.731704] Lustre: server umount lustre-MDT0000 complete [10577.888961] LustreError: 510062:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10577.895305] LustreError: 510062:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10578.231152] LustreError: 510042:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767325545 with bad export cookie 11431637541420309649 [10578.234382] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10578.235462] LustreError: 510042:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [10578.236182] LustreError: 510976:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000bd2:0x1:0x0]: rc = -5 [10578.349729] Lustre: server umount lustre-MDT0001 complete [10582.592650] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [10583.623576] Key type lgssc unregistered [10583.763392] LNet: 513891:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10583.766697] LNetError: 513891:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [10583.776335] LNet: Removed LNI 192.168.203.152@tcp [10584.097888] Key type .llcrypt unregistered [10584.098989] Key type ._llcrypt unregistered [10599.148471] Key type ._llcrypt registered [10599.150674] Key type .llcrypt registered [10599.190600] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_hostid [10623.119983] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [10623.631820] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10623.645559] alg: No test for adler32 (adler32-zlib) [10624.507600] Lustre: Lustre: Build Version: 2.17.0_RC4_1_g3742a8b [10624.595361] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [10626.176151] Key type lgssc registered [10626.550833] Lustre: Echo OBD driver; http://www.lustre.org/ [10632.814776] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10636.649649] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10639.856425] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10643.138378] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10645.411153] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10647.757920] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10652.920148] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10652.937896] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10652.943675] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10654.034265] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10654.045969] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10654.089529] Lustre: lustre-MDT0000: new disk, initializing [10654.119106] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10654.126876] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10655.395832] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [10661.620414] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10661.643621] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10661.676260] Lustre: 523566:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10661.703628] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10661.705922] Lustre: Skipped 1 previous similar message [10661.746348] Lustre: lustre-MDT0001: new disk, initializing [10661.788387] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10661.805139] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10661.810844] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10663.148542] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [10666.260371] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10667.404224] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10671.652362] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10671.678463] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10671.771312] Lustre: lustre-OST2710: new disk, initializing [10671.773047] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [10671.804173] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [10673.681886] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [10676.784265] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [10678.270624] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [10678.274907] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [10678.319444] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [10681.203172] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10681.226932] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10681.278520] Lustre: lustre-OST03e8: new disk, initializing [10681.280900] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [10681.308858] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [10683.244549] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [10685.089659] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [10686.982094] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [10686.985816] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [10687.019891] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [10687.647902] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [10687.720408] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10689.069368] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [10689.140546] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10690.497797] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [10690.565796] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10691.874884] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [10691.941757] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10697.190482] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10697.192628] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10697.194253] Lustre: Skipped 3 previous similar messages [10697.196676] Lustre: Skipped 2 previous similar messages [10699.074922] LustreError: 528944:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [10699.109380] Lustre: server umount lustre-MDT0000 complete [10700.328286] LustreError: 523558:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767325667 with bad export cookie 16011459445487061677 [10700.332732] LustreError: 523558:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [10700.332800] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10700.403246] LustreError: 529146:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [10700.405946] LustreError: 529146:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [10700.453243] Lustre: server umount lustre-MDT0001 complete [10713.013080] LustreError: 529491:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [10713.015491] LustreError: 529491:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [10713.036358] Lustre: server umount lustre-OST2710 complete [10724.594379] LustreError: 529693:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [10724.596154] LustreError: 529693:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [10724.633993] Lustre: server umount lustre-OST03e8 complete [10738.331256] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_hostid [10758.434738] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [10765.066867] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10768.695101] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10771.641844] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10774.889283] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10780.016142] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10780.036414] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10780.117760] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10780.129385] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10780.170223] Lustre: lustre-MDT0000: new disk, initializing [10780.194520] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10780.201730] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10781.493413] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [10787.491509] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10787.513160] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10787.545920] Lustre: 538534:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10787.641937] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10788.868537] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [10791.909814] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10793.051723] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10797.348316] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10797.372916] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10797.471145] Lustre: lustre-OST0000: new disk, initializing [10797.472894] Lustre: Skipped 1 previous similar message [10797.475179] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10797.477391] Lustre: Skipped 2 previous similar messages [10797.508223] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10797.510590] Lustre: Skipped 1 previous similar message [10799.305883] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [10799.352464] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10799.357074] Lustre: Skipped 1 previous similar message [10799.358547] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10799.394291] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10802.371414] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10803.689850] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10803.761108] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10804.991360] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10805.072973] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10809.826492] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10809.826932] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10809.828540] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10809.828544] Lustre: Skipped 1 previous similar message [10809.829587] Lustre: Skipped 1 previous similar message [10809.835283] LustreError: Skipped 1 previous similar message [10814.946421] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10814.948849] Lustre: Skipped 2 previous similar messages [10820.064146] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10820.067770] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10820.070454] Lustre: Skipped 1 previous similar message [10820.083078] LustreError: 542162:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [10820.085916] LustreError: 542162:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [10820.127700] Lustre: server umount lustre-OST0000 complete [10823.648772] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10823.650757] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10825.185278] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10825.186943] Lustre: Skipped 2 previous similar messages [10827.862291] Lustre: server umount lustre-MDT0000 complete [10829.362018] LustreError: 542414:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767325796 with bad export cookie 16011459445487063350 [10829.364553] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10829.365366] LustreError: 542414:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [10829.432319] LustreError: 542664:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [10829.434122] LustreError: 542664:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [10829.484289] Lustre: server umount lustre-MDT0001 complete [10832.175752] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 22:49:59 (1767325799) [10832.658232] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [10833.186256] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 22:50:00 (1767325800) [10834.123567] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [10837.065615] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10838.562922] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10840.159924] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10841.728505] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10842.377544] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10846.510239] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10846.607328] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10846.614440] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10846.674991] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10847.973244] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [10851.788378] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10851.828335] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10851.837974] Lustre: 545364:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10853.225198] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [10854.905397] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10856.002172] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10859.266355] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10859.290233] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10859.361099] Lustre: Denying initial registration attempt from nid 192.168.203.152@tcp, specified as failover [10859.363729] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [10859.366270] LustreError: 547364:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -99 [10859.368289] LustreError: 547364:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [10859.370412] LustreError: 547364:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [10859.391592] Lustre: server umount lustre-OST0000 complete [10859.393525] LustreError: 547364:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -99 [10862.048928] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10862.052292] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10862.055440] Lustre: Skipped 1 previous similar message [10862.057732] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10876.384149] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10876.425447] LustreError: 547779:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [10876.428028] LustreError: 547779:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [10876.452832] Lustre: server umount lustre-MDT0000 complete [10877.962908] LustreError: 545354:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767325845 with bad export cookie 16011459445487064232 [10877.965825] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10877.965921] LustreError: 545354:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [10878.081525] Lustre: server umount lustre-MDT0001 complete [10882.259062] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [10883.273332] Key type lgssc unregistered [10883.406322] LNet: 548707:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10883.409309] LNetError: 548707:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [10883.416285] LNet: Removed LNI 192.168.203.152@tcp [10883.719099] Key type .llcrypt unregistered [10883.719916] Key type ._llcrypt unregistered [10887.178465] Key type ._llcrypt registered [10887.179757] Key type .llcrypt registered [10887.216287] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [10887.544967] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10887.553933] alg: No test for adler32 (adler32-zlib) [10888.399281] Lustre: Lustre: Build Version: 2.17.0_RC4_1_g3742a8b [10888.481307] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [10890.056108] Key type lgssc registered [10890.431413] Lustre: Echo OBD driver; http://www.lustre.org/ [10892.948612] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10894.459326] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10895.908897] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10897.390856] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10898.011851] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10922.907566] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [10925.726912] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10925.736431] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10926.829309] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10926.839255] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10926.904497] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10928.200638] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [10932.162965] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10932.193802] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10932.205337] Lustre: 554896:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10932.292068] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10933.565463] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [10935.310881] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10936.417379] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10939.656261] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10939.680541] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10939.744860] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10939.806618] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10941.550631] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [10943.192638] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10945.506115] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10945.507808] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10945.509401] LustreError: Skipped 1 previous similar message [10945.515139] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10945.517662] Lustre: Skipped 1 previous similar message [10949.933685] LustreError: 557562:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [10949.970356] Lustre: server umount lustre-OST0000 complete [10952.672755] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10952.674668] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10952.677929] Lustre: Skipped 1 previous similar message [10952.679940] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10955.745193] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10955.745690] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10957.502821] LustreError: 557812:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [10957.504770] LustreError: 557812:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [10957.530607] Lustre: server umount lustre-MDT0000 complete [10959.079477] LustreError: 554887:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767325926 with bad export cookie 7413095594533154486 [10959.082422] LustreError: 554887:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [10959.083213] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10959.153493] LustreError: 558062:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [10959.156170] LustreError: 558062:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [10959.219274] Lustre: server umount lustre-MDT0001 complete [10963.473726] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [10964.458893] Key type lgssc unregistered [10964.591552] LNet: 558738:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10964.594443] LNetError: 558738:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [10964.603245] LNet: Removed LNI 192.168.203.152@tcp [10964.880672] Key type .llcrypt unregistered [10964.882127] Key type ._llcrypt unregistered [10993.111690] Key type ._llcrypt registered [10993.112703] Key type .llcrypt registered [10993.152166] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [10993.491482] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10993.588782] alg: No test for adler32 (adler32-zlib) [10994.423364] Lustre: Lustre: Build Version: 2.17.0_RC4_1_g3742a8b [10994.505923] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [10996.088154] Key type lgssc registered [10996.458138] Lustre: Echo OBD driver; http://www.lustre.org/ [10998.881796] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10998.888259] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11000.048289] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11001.269793] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [11005.012676] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11005.128707] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11006.366333] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [11008.000646] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11009.152598] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11012.385629] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11012.501888] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11014.251616] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [11015.867727] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11031.010812] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11031.011544] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11031.014716] Lustre: Skipped 2 previous similar messages [11031.015969] Lustre: Skipped 1 previous similar message [11031.019793] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11034.431818] LustreError: 566174:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [11034.457095] Lustre: server umount lustre-MDT0000 complete [11035.887954] LustreError: 563455:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767326003 with bad export cookie 3416897165082353882 [11035.890644] LustreError: 563455:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [11035.891869] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11036.128748] LustreError: 565961:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11036.128996] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11036.132684] LustreError: 565961:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11036.137637] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11036.141069] Lustre: Skipped 2 previous similar messages [11040.160753] LustreError: 563468:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11040.160980] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11040.166587] LustreError: 563468:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11040.168200] Lustre: Skipped 1 previous similar message [11041.613678] LustreError: 563468:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.52@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11042.042861] LustreError: 566425:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [11042.044628] LustreError: 566425:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [11042.084994] Lustre: server umount lustre-MDT0001 complete [11044.636313] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [11048.702756] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11061.664867] LustreError: 562783:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000002b81888 x1853175586904448/t0(0) o250->MGC192.168.203.152@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11061.785716] LustreError: 565465:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11061.807854] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11061.826763] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [11063.074755] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [11065.889433] LustreError: 567572:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11065.893481] LustreError: 567572:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11066.773899] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11066.892054] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11068.165309] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [11069.849223] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11069.989995] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [11073.403949] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [11074.591402] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11075.154812] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [11077.089898] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11077.090338] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11077.092821] LustreError: Skipped 1 previous similar message [11077.097335] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11077.100259] Lustre: Skipped 2 previous similar messages [11082.481072] LustreError: 569255:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [11082.483782] LustreError: 569255:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [11082.526954] Lustre: server umount lustre-OST0000 complete [11085.282797] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11085.283505] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11085.286369] Lustre: Skipped 2 previous similar messages [11085.288332] Lustre: Skipped 2 previous similar messages [11090.237965] LustreError: 569505:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [11090.239975] LustreError: 569505:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [11090.269877] Lustre: server umount lustre-MDT0000 complete [11090.401079] LustreError: 567576:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11090.406119] LustreError: 567576:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11091.787289] LustreError: 565475:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767326058 with bad export cookie 3416897165082361036 [11091.790949] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11091.791696] LustreError: 565475:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [11091.913158] Lustre: server umount lustre-MDT0001 complete [11096.255547] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [11097.286816] Key type lgssc unregistered [11097.410445] LNet: 570431:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11097.413202] LNetError: 570431:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [11097.422324] LNet: Removed LNI 192.168.203.152@tcp [11097.738063] Key type .llcrypt unregistered [11097.739280] Key type ._llcrypt unregistered [11125.721718] Key type ._llcrypt registered [11125.723097] Key type .llcrypt registered [11125.762708] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [11126.218975] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11126.226619] alg: No test for adler32 (adler32-zlib) [11127.094696] Lustre: Lustre: Build Version: 2.17.0_RC4_1_g3742a8b [11127.185204] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [11128.768155] Key type lgssc registered [11129.160317] Lustre: Echo OBD driver; http://www.lustre.org/ [11131.859978] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11131.867288] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11133.026518] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11134.422664] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [11138.719611] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11138.876348] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11140.197903] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [11142.020860] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11143.193819] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11148.257793] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11148.258424] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11148.261914] Lustre: Skipped 1 previous similar message [11148.265034] Lustre: Skipped 1 previous similar message [11149.281624] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11149.283729] Lustre: Skipped 1 previous similar message [11150.336350] LustreError: 576609:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [11150.367123] Lustre: server umount lustre-MDT0000 complete [11151.646724] LustreError: 575937:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767326118 with bad export cookie 9138394466929683225 [11151.650064] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11151.650664] LustreError: 575937:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [11151.727296] LustreError: 576811:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [11151.729103] LustreError: 576811:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [11151.785315] Lustre: server umount lustre-MDT0001 complete [11160.649328] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11160.749137] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [11160.759030] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11160.843427] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11162.201374] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [11166.259707] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11166.294099] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11166.304609] Lustre: 578205:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11166.407160] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11167.776443] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [11169.557279] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11170.745576] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11174.481641] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11174.551937] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [11174.556746] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [11174.559504] LustreError: 580247:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -2 [11174.561612] LustreError: 580247:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [11174.563432] LustreError: 580247:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [11174.587163] Lustre: server umount lustre-OST0000 complete [11174.588552] LustreError: 580247:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -2 [11178.585161] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11178.652695] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [11178.716642] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11180.523687] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [11182.260759] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11184.672867] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [11186.980712] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11187.003535] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11187.026324] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [11187.055624] Lustre: lustre-OST0001: new disk, initializing [11187.057520] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11187.087377] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11188.993808] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [11190.777925] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [11191.770118] LustreError: 582891:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [11191.772324] LustreError: 582891:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [11191.790473] Lustre: server umount lustre-OST0001 complete [11192.827175] LustreError: 580905:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11192.832186] LustreError: 580905:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11194.850303] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11194.850513] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11194.853234] LustreError: Skipped 1 previous similar message [11194.857338] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11194.860234] Lustre: Skipped 1 previous similar message [11197.924613] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11197.925556] LustreError: 581818:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11197.927090] Lustre: Skipped 1 previous similar message [11197.931449] LustreError: 581818:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11199.086507] LustreError: 583102:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [11199.088958] LustreError: 583102:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [11199.124545] Lustre: server umount lustre-OST0000 complete [11202.528818] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11202.531186] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11202.534695] Lustre: Skipped 1 previous similar message [11202.536351] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11206.784456] LustreError: 583352:0:(obd_class.h:479:obd_check_dev()) Device 24 not setup [11206.786563] LustreError: 583352:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [11206.818158] Lustre: server umount lustre-MDT0000 complete [11208.162190] LustreError: 579128:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11208.167852] LustreError: 579128:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11208.391547] LustreError: 579122:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767326175 with bad export cookie 9138394466929683764 [11208.397888] LustreError: 579122:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [11208.398317] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11208.549533] Lustre: server umount lustre-MDT0001 complete [11213.185756] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [11214.162167] Key type lgssc unregistered [11214.279328] LNet: 584278:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11214.281969] LNetError: 584278:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [11214.290310] LNet: Removed LNI 192.168.203.152@tcp [11214.593704] Key type .llcrypt unregistered [11214.595250] Key type ._llcrypt unregistered [11234.886584] Key type ._llcrypt registered [11234.887757] Key type .llcrypt registered [11234.928317] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_hostid [11259.212652] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [11259.688490] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11259.750791] alg: No test for adler32 (adler32-zlib) [11260.623180] Lustre: Lustre: Build Version: 2.17.0_RC4_1_g3742a8b [11260.718807] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [11262.304157] Key type lgssc registered [11262.738601] Lustre: Echo OBD driver; http://www.lustre.org/ [11269.391230] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11273.233842] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11276.459335] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11279.709535] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11284.583047] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11284.606606] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11284.612884] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11285.708947] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11285.720888] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11285.762806] Lustre: lustre-MDT0000: new disk, initializing [11285.788163] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11285.794577] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11287.056583] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [11293.322256] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11293.346920] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11293.380377] Lustre: 594296:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11293.405140] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11293.406891] Lustre: Skipped 1 previous similar message [11293.444500] Lustre: lustre-MDT0001: new disk, initializing [11293.476348] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11293.488777] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11293.491861] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11294.811057] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [11297.850605] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11298.963372] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11303.327553] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11303.351910] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11303.461158] Lustre: lustre-OST0000: new disk, initializing [11303.463937] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11303.493828] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11305.359661] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [11308.403304] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11309.763504] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11311.095474] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11311.099615] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11311.135439] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11311.882682] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [11313.234176] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11313.308877] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11316.194909] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11316.195130] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11316.199394] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11316.202355] Lustre: Skipped 1 previous similar message [11320.048071] LustreError: 597940:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [11320.090596] Lustre: server umount lustre-OST0000 complete [11324.385095] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11324.387226] LustreError: Skipped 1 previous similar message [11324.388416] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11324.393416] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11324.394991] Lustre: Skipped 1 previous similar message [11326.433209] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11326.433789] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11327.810779] LustreError: 598189:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [11327.812592] LustreError: 598189:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [11327.841752] Lustre: server umount lustre-MDT0000 complete [11329.354256] LustreError: 594286:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767326296 with bad export cookie 3912140612538721003 [11329.358448] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11329.360104] LustreError: 594286:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [11329.426325] LustreError: 598440:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [11329.428280] LustreError: 598440:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [11329.484905] Lustre: server umount lustre-MDT0001 complete [11332.170395] Lustre: DEBUG MARKER: == conf-sanity test 60a: check mkfs.lustre --mkfsoptions -E -O options setting ========================================================== 22:58:19 (1767326299) [11336.092752] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11339.078050] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11353.390220] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_hostid [11374.342577] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [11380.800440] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11384.544787] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11387.695706] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11390.998043] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11396.090285] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11396.114692] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11396.207488] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11396.220091] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11396.260961] Lustre: lustre-MDT0000: new disk, initializing [11396.284913] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11396.292388] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11397.543731] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [11403.817451] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11403.839723] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11403.873716] Lustre: 608586:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11403.902201] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11403.904183] Lustre: Skipped 1 previous similar message [11403.941439] Lustre: lustre-MDT0001: new disk, initializing [11403.975003] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11403.988562] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11403.992071] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11405.231864] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [11408.326561] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11409.462680] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11413.711451] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11413.732558] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11413.836133] Lustre: lustre-OST0000: new disk, initializing [11413.838139] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11413.865788] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11415.714268] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [11418.749710] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11419.641559] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11419.646134] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11419.682868] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11420.134263] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11420.210184] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11421.526974] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11421.602153] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11424.739641] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11424.740428] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11424.741772] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11424.741778] Lustre: Skipped 1 previous similar message [11424.742357] LustreError: Skipped 1 previous similar message [11424.745953] Lustre: Skipped 1 previous similar message [11428.270142] LustreError: 612212:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [11428.272315] LustreError: 612212:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [11428.308132] Lustre: server umount lustre-OST0000 complete [11434.978070] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11434.978136] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11434.978720] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11434.978741] Lustre: Skipped 1 previous similar message [11434.986719] Lustre: Skipped 1 previous similar message [11436.224053] LustreError: 612462:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [11436.225915] LustreError: 612462:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [11436.255690] Lustre: server umount lustre-MDT0000 complete [11437.799337] LustreError: 609710:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767326404 with bad export cookie 3912140612538721885 [11437.802416] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11437.805196] LustreError: 609710:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [11437.910197] Lustre: server umount lustre-MDT0001 complete [11440.754112] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 23:00:07 (1767326407) [11443.408752] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 23:00:10 (1767326410) [11448.083400] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11448.250800] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11449.491429] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [11453.409415] LustreError: 614060:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11453.415198] LustreError: 614060:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11453.560283] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11455.001971] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [11456.746716] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11457.913834] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11461.257483] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11463.190484] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [11464.911766] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11478.678259] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [11479.276832] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [11479.836571] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [11480.397150] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [11481.119154] Lustre: Failing over lustre-MDT0000 [11481.303755] LustreError: 616903:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [11481.305299] LustreError: 616903:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [11481.333072] Lustre: server umount lustre-MDT0000 complete [11484.131026] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11484.132237] LustreError: 614055:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11484.134891] Lustre: Skipped 2 previous similar messages [11484.139030] LustreError: 614055:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11485.523460] LustreError: 614054:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.52@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11485.528070] LustreError: 614054:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11489.249707] LustreError: 615625:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11489.254944] LustreError: 615625:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11494.370990] LustreError: 615625:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11494.380721] LustreError: 615625:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [11497.992512] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11498.033431] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11498.127516] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11498.129510] Lustre: Skipped 2 previous similar messages [11498.156571] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [11499.469781] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [11500.878536] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [11503.595894] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11503.604753] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [11503.624619] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [11504.813569] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11505.341453] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11507.213336] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [11507.841190] Lustre: Failing over lustre-MDT0000 [11507.979815] LustreError: 618572:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [11507.982350] LustreError: 618572:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [11508.010873] Lustre: server umount lustre-MDT0000 complete [11508.706614] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11508.707072] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11508.707967] LustreError: 614084:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11508.707974] LustreError: 614084:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11508.725867] Lustre: Skipped 1 previous similar message [11524.064075] Lustre: 590771:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767326475/real 1767326475] req@00000000203a33df x1853175865900928/t0(0) o400->MGC192.168.203.152@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1767326491 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11524.071293] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11524.252334] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11526.478361] LustreError: 614084:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.52@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11526.483315] LustreError: 614084:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 14 previous similar messages [11534.307308] LustreError: 590770:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000e521727b x1853175865906560/t0(0) o250->MGC192.168.203.152@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11534.428857] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [11535.195612] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [11535.745027] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [11539.429469] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11539.432815] Lustre: Skipped 2 previous similar messages [11539.440762] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [11539.461101] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [11540.627774] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11541.205459] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11544.546582] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11544.547208] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11544.549268] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11544.549275] Lustre: Skipped 1 previous similar message [11544.549793] Lustre: Skipped 3 previous similar messages [11549.566791] LustreError: 620158:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [11549.568732] LustreError: 620158:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [11549.593927] Lustre: server umount lustre-MDT0000 complete [11550.899305] LustreError: 614039:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767326518 with bad export cookie 3912140612538724573 [11550.905218] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11551.021348] Lustre: server umount lustre-MDT0001 complete [11563.487334] Lustre: server umount lustre-OST0000 complete [11572.154958] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11572.317077] LustreError: 622107:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11572.323048] LustreError: 622107:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 10 previous similar messages [11572.340198] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11572.342739] Lustre: Skipped 1 previous similar message [11573.730570] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [11577.845378] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11579.299989] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [11581.097654] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11582.255409] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11585.619230] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11587.560478] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [11589.285935] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11597.031073] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [11602.405874] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11602.406973] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11602.412076] Lustre: Skipped 3 previous similar messages [11606.991079] Lustre: server umount lustre-OST0000 complete [11612.641576] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11612.645140] Lustre: Skipped 3 previous similar messages [11614.785256] LustreError: 625009:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [11614.788224] LustreError: 625009:0:(obd_class.h:479:obd_check_dev()) Skipped 21 previous similar messages [11614.817896] Lustre: server umount lustre-MDT0000 complete [11616.337196] LustreError: 625011:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767326583 with bad export cookie 3912140612538725301 [11616.340034] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11616.342164] LustreError: 625011:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 5 previous similar messages [11616.469568] Lustre: server umount lustre-MDT0001 complete [11621.069722] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [11622.116248] Key type lgssc unregistered [11622.237362] LNet: 625935:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11622.239833] LNetError: 625935:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [11622.246286] LNet: Removed LNI 192.168.203.152@tcp [11622.534137] Key type .llcrypt unregistered [11622.535660] Key type ._llcrypt unregistered [11652.014681] Key type ._llcrypt registered [11652.016376] Key type .llcrypt registered [11652.058415] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [11652.409790] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11652.466643] alg: No test for adler32 (adler32-zlib) [11653.315106] Lustre: Lustre: Build Version: 2.17.0_RC4_1_g3742a8b [11653.397762] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [11654.976156] Key type lgssc registered [11655.299767] Lustre: Echo OBD driver; http://www.lustre.org/ [11657.823791] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11657.834637] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11659.006547] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11660.238295] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [11664.076169] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11664.195033] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11665.445356] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [11667.173436] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11668.384237] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11671.737526] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11671.868313] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11673.659492] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [11675.401500] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11684.205635] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [11689.953232] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11689.957640] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11689.964345] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11694.561447] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11694.563626] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11696.060430] LustreError: 633277:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [11696.089984] Lustre: server umount lustre-MDT0000 complete [11697.234094] LustreError: 631488:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767326664 with bad export cookie 15391738019773212050 [11697.237311] LustreError: 631488:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [11697.237417] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11697.298540] LustreError: 633480:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [11697.300974] LustreError: 633480:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [11697.343752] Lustre: server umount lustre-MDT0001 complete [11709.684785] LustreError: 633824:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [11709.686403] LustreError: 633824:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [11709.739371] Lustre: server umount lustre-OST0000 complete [11718.768283] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11718.921440] LustreError: 635035:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11718.926765] LustreError: 635035:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11718.947697] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11720.437653] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [11724.257065] LustreError: 635035:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11724.424085] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11724.535682] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11725.865928] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [11727.595186] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11728.724312] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11732.227256] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11734.047074] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [11735.790276] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11744.617037] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [11754.978767] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11754.980443] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11754.982725] Lustre: Skipped 3 previous similar messages [11754.984978] Lustre: Skipped 3 previous similar messages [11758.911916] LustreError: 637788:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [11758.913640] LustreError: 637788:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [11758.939185] Lustre: server umount lustre-MDT0000 complete [11760.076039] LustreError: 637702:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767326727 with bad export cookie 15391738019773213149 [11760.079037] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11760.081419] LustreError: 637702:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [11760.096888] LustreError: 635030:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11760.097169] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11760.106551] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11760.108184] Lustre: Skipped 1 previous similar message [11760.186275] Lustre: server umount lustre-MDT0001 complete [11768.366083] LustreError: 638334:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [11768.369751] LustreError: 638334:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [11768.411420] Lustre: server umount lustre-OST0000 complete [11776.721061] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11776.868514] LustreError: 639738:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11776.891302] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11776.894279] Lustre: Skipped 1 previous similar message [11778.175182] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [11782.044319] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11782.112905] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [11783.585166] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [11785.334949] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11786.552260] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11789.893838] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11791.816819] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [11793.499616] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11801.255404] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [11806.691398] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11806.694329] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11806.696142] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11806.700388] Lustre: Skipped 1 previous similar message [11811.247113] LustreError: 642387:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [11811.248988] LustreError: 642387:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [11811.285397] Lustre: server umount lustre-OST0000 complete [11812.832674] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11816.929866] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11816.935214] Lustre: Skipped 1 previous similar message [11826.145365] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11826.147818] Lustre: Skipped 7 previous similar messages [11827.168113] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11827.229860] Lustre: server umount lustre-MDT0000 complete [11828.694237] LustreError: 639718:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767326795 with bad export cookie 15391738019773214157 [11828.697430] LustreError: 639718:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [11828.697621] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11828.765086] LustreError: 642891:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [11828.768330] LustreError: 642891:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [11828.817952] Lustre: server umount lustre-MDT0001 complete [11833.207928] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [11834.133978] Key type lgssc unregistered [11834.253321] LNet: 643567:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11834.256174] LNetError: 643567:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [11834.265342] LNet: Removed LNI 192.168.203.152@tcp [11834.542712] Key type .llcrypt unregistered [11834.543815] Key type ._llcrypt unregistered [11862.784630] Key type ._llcrypt registered [11862.785859] Key type .llcrypt registered [11862.823570] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [11863.114385] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11863.123666] alg: No test for adler32 (adler32-zlib) [11863.969832] Lustre: Lustre: Build Version: 2.17.0_RC4_1_g3742a8b [11864.046710] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [11865.624107] Key type lgssc registered [11865.955212] Lustre: Echo OBD driver; http://www.lustre.org/ [11868.383182] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11868.388396] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11868.391122] LustreError: 648139:0:(osd_handler.c:8561:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [11868.398712] LustreError: 648139:0:(obd_config.c:783:class_setup()) setup lustre-MDT0000-osd failed (-22) [11868.400886] LustreError: 648139:0:(obd_mount.c:193:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [11868.402992] LustreError: 648139:0:(tgt_mount.c:2447:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [11868.406584] LustreError: 648139:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -22 [11871.867127] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: user_xattr,acl,no_mbcache,nodelalloc [11871.869628] LustreError: 648728:0:(osd_handler.c:8561:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [11871.876764] LustreError: 648728:0:(obd_config.c:783:class_setup()) setup lustre-OST0000-osd failed (-22) [11871.878841] LustreError: 648728:0:(obd_mount.c:193:lustre_start_simple()) lustre-OST0000-osd setup error -22 [11871.881068] LustreError: 648728:0:(tgt_mount.c:2447:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [11871.883465] LustreError: 648728:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -22 [11879.647124] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [11880.709370] Key type lgssc unregistered [11880.840371] LNet: 649953:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11880.843687] LNetError: 649953:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [11880.857290] LNet: Removed LNI 192.168.203.152@tcp [11881.167415] Key type .llcrypt unregistered [11881.168771] Key type ._llcrypt unregistered [11893.543833] Key type ._llcrypt registered [11893.545042] Key type .llcrypt registered [11893.582871] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_hostid [11916.995905] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [11917.526873] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11917.541123] alg: No test for adler32 (adler32-zlib) [11918.405159] Lustre: Lustre: Build Version: 2.17.0_RC4_1_g3742a8b [11918.493319] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [11920.072154] Key type lgssc registered [11920.472727] Lustre: Echo OBD driver; http://www.lustre.org/ [11926.925047] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11930.784881] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11933.925912] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11937.322879] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11942.426775] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11942.448799] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11942.453181] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11943.533725] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11943.545354] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11943.581480] Lustre: lustre-MDT0000: new disk, initializing [11943.604479] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11943.611203] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11944.847115] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [11950.824713] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11950.847058] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11950.879208] Lustre: 658761:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11950.902148] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11950.903844] Lustre: Skipped 1 previous similar message [11950.937450] Lustre: lustre-MDT0001: new disk, initializing [11950.966691] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11950.978500] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11950.981656] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11952.248295] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [11955.212736] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11956.642252] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11961.122731] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11961.152553] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11961.245892] Lustre: lustre-OST0000: new disk, initializing [11961.248107] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11961.273938] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11963.100642] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [11965.238386] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11965.240933] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11965.269713] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11966.295662] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11967.615253] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11967.683663] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11968.903167] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11968.968147] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11970.530213] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11970.531287] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11970.532465] LustreError: Skipped 1 previous similar message [11970.538950] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11970.541940] Lustre: Skipped 1 previous similar message [11971.630090] LustreError: 662391:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [11971.665638] Lustre: server umount lustre-OST0000 complete [11975.649697] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11975.650626] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11975.653112] Lustre: Skipped 2 previous similar messages [11979.264965] LustreError: 662641:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [11979.266927] LustreError: 662641:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [11979.292730] Lustre: server umount lustre-MDT0000 complete [11981.166269] LustreError: 659896:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767326948 with bad export cookie 6887693990927307695 [11981.169230] LustreError: 659896:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [11981.169705] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11981.230286] LustreError: 662890:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [11981.231939] LustreError: 662890:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [11981.282759] Lustre: server umount lustre-MDT0001 complete [11983.911952] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [11984.426734] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 23:09:11 (1767326951) [11987.735454] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11987.893625] LustreError: 663800:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11987.900104] LustreError: 663800:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [11987.915935] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11989.153593] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [11993.057025] LustreError: 663800:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11993.148154] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11993.254970] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11994.508426] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [11996.158429] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11997.310185] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12000.660932] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12002.571989] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [12004.327101] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12008.610326] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12008.638228] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12008.695050] Lustre: lustre-OST0001: new disk, initializing [12008.696639] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [12008.722703] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12008.724555] Lustre: Skipped 1 previous similar message [12009.865182] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [12009.868087] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [12009.911189] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [12010.596093] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [12013.645143] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [12018.149222] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12018.150073] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [12018.152752] Lustre: Skipped 1 previous similar message [12018.154224] Lustre: Skipped 1 previous similar message [12021.551074] LustreError: 667726:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [12021.553528] LustreError: 667726:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [12021.570788] Lustre: server umount lustre-OST0001 complete [12023.268900] LustreError: 667230:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12023.275679] LustreError: 667230:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12028.388414] LustreError: 665780:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12028.392731] LustreError: 665780:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12033.507602] LustreError: 665780:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12033.512844] LustreError: 665780:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12034.530383] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12034.532345] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12034.535649] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12034.537199] Lustre: Skipped 2 previous similar messages [12038.627163] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12038.628429] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12038.629296] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12039.597119] LustreError: 667949:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [12039.600472] LustreError: 667949:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [12039.633046] Lustre: server umount lustre-OST0000 complete [12047.382147] Lustre: server umount lustre-MDT0000 complete [12048.804987] LustreError: 663781:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767327015 with bad export cookie 6887693990927308577 [12048.808095] LustreError: 663781:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [12048.808539] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12063.200175] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12063.236101] LustreError: 668449:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [12063.238141] LustreError: 668449:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [12063.294621] Lustre: server umount lustre-MDT0001 complete [12067.750293] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [12068.691858] Key type lgssc unregistered [12068.807290] LNet: 669126:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12068.809580] LNetError: 669126:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [12068.819348] LNet: Removed LNI 192.168.203.152@tcp [12069.104963] Key type .llcrypt unregistered [12069.106101] Key type ._llcrypt unregistered [12088.534792] Key type ._llcrypt registered [12088.535780] Key type .llcrypt registered [12088.572512] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_hostid [12111.710807] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [12112.129531] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12112.138570] alg: No test for adler32 (adler32-zlib) [12112.985467] Lustre: Lustre: Build Version: 2.17.0_RC4_1_g3742a8b [12113.059779] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [12114.640112] Key type lgssc registered [12114.979179] Lustre: Echo OBD driver; http://www.lustre.org/ [12121.053059] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12124.742524] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12127.748805] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12130.918510] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12135.843398] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12135.860899] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12135.865365] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12136.943563] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12136.953521] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12136.989322] Lustre: lustre-MDT0000: new disk, initializing [12137.011887] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12137.018175] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12138.223667] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [12144.164161] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12144.184856] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12144.216784] Lustre: 679144:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12144.237319] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12144.239869] Lustre: Skipped 1 previous similar message [12144.275818] Lustre: lustre-MDT0001: new disk, initializing [12144.306245] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12144.316727] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12144.320233] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12145.517973] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [12148.499793] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12149.711204] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12154.095304] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12154.120906] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12154.235982] Lustre: lustre-OST0000: new disk, initializing [12154.237766] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12154.274120] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12155.264103] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12155.267328] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12155.308383] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12156.101383] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [12159.121351] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12160.427462] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12160.496469] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12161.768818] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12161.843284] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12165.602681] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12165.603055] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12165.607230] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12165.608407] LustreError: Skipped 1 previous similar message [12165.611406] Lustre: Skipped 1 previous similar message [12168.493174] LustreError: 682775:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [12168.531528] Lustre: server umount lustre-OST0000 complete [12170.209177] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12170.212993] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12170.220155] Lustre: Skipped 1 previous similar message [12170.223593] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12175.842177] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12175.844458] Lustre: Skipped 2 previous similar messages [12176.199185] LustreError: 683025:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [12176.202281] LustreError: 683025:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [12176.233262] Lustre: server umount lustre-MDT0000 complete [12177.806086] LustreError: 680271:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767327144 with bad export cookie 4279187927515629977 [12177.811585] LustreError: 680271:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [12177.811935] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12177.881118] LustreError: 683275:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [12177.884541] LustreError: 683275:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [12177.938790] Lustre: server umount lustre-MDT0001 complete [12180.743446] Lustre: DEBUG MARKER: == conf-sanity test 65: re-create the lost last_rcvd file when server mount ========================================================== 23:12:27 (1767327147) [12184.675650] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [12188.524214] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12188.667222] Lustre: lustre-MDT0000: new disk, initializing [12188.698097] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12189.942498] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [12193.720972] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12193.760870] LustreError: 684878:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12193.765472] LustreError: 684878:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12193.834550] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12195.037781] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [12196.720659] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12197.857303] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12198.880872] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12198.883175] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12198.886808] Lustre: Skipped 1 previous similar message [12198.888525] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12202.977243] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12202.977817] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12213.216078] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12213.217154] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12213.221829] Lustre: Skipped 4 previous similar messages [12213.255314] LustreError: 686462:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [12213.258555] LustreError: 686462:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [12213.286874] Lustre: server umount lustre-MDT0000 complete [12214.739041] LustreError: 684858:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767327181 with bad export cookie 4279187927515630852 [12214.741906] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12214.742245] LustreError: 684858:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [12214.849719] Lustre: server umount lustre-MDT0001 complete [12217.782986] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 23:13:04 (1767327184) [12220.956263] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12221.093436] LustreError: 687624:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12221.114255] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12222.355078] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [12226.127636] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12227.425534] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [12229.084303] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12230.247937] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12233.455683] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12235.275859] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [12236.950235] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12250.017328] Lustre: Permanently deactivating lustre-OST0000 [12250.022117] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [12258.296482] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12259.622470] LustreError: 690819:0:(mgs_llog.c:1942:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [12259.626145] LustreError: 690819:0:(mgs_llog.c:1997:mgs_replace_nids()) Only MGS is allowed to be started [12259.627917] LustreError: 690819:0:(mgs_handler.c:1171:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12260.200254] LustreError: 690868:0:(mgs_llog.c:1997:mgs_replace_nids()) Only MGS is allowed to be started [12260.202578] LustreError: 690868:0:(mgs_handler.c:1171:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12266.863598] LustreError: 690965:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [12266.865507] LustreError: 690965:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [12266.898764] Lustre: server umount lustre-OST0000 complete [12267.737283] LustreError: 691117:0:(mgs_llog.c:1997:mgs_replace_nids()) Only MGS is allowed to be started [12267.739549] LustreError: 691117:0:(mgs_handler.c:1171:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12272.609955] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12272.610019] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12272.610740] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12272.613944] Lustre: Skipped 1 previous similar message [12278.756183] LustreError: 691262:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [12278.758587] LustreError: 691262:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [12278.778744] Lustre: server umount lustre-MDT0000 complete [12280.268423] LustreError: 690379:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767327247 with bad export cookie 4279187927515631391 [12280.272019] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12280.273851] LustreError: 690379:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [12290.569071] Lustre: server umount lustre-MDT0001 complete [12295.274907] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12296.551571] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [12297.425130] LustreError: 692691:0:(mgs_handler.c:1171:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [12299.441991] Lustre: 693029:0:(mgs_llog.c:1709:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.152@tcp [12301.311932] LustreError: 693318:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [12301.313643] LustreError: 693318:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [12301.333978] Lustre: server umount MGS complete [12307.217026] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12307.380399] LustreError: 694331:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12307.385795] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12307.388839] Lustre: Skipped 1 previous similar message [12307.408047] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12307.409952] Lustre: Skipped 2 previous similar messages [12308.670402] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [12312.545517] LustreError: 694331:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12312.554047] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12312.650410] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12313.915259] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [12315.573634] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12316.784829] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12317.410536] Lustre: Permanently reactivating lustre-OST0000 [12317.416743] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [12317.419613] Lustre: Skipped 2 previous similar messages [12341.728140] LustreError: 696038:0:(osp_dev.c:692:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [12345.297064] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12345.415211] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12345.418306] Lustre: Skipped 1 previous similar message [12347.243275] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [12349.006482] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12362.210682] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12362.211327] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12362.213511] LustreError: Skipped 1 previous similar message [12362.219852] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12362.222278] Lustre: Skipped 4 previous similar messages [12362.799093] LustreError: 698051:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [12362.802100] LustreError: 698051:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [12362.844220] Lustre: server umount lustre-OST0000 complete [12366.438353] Lustre: server umount lustre-MDT0000 complete [12367.330349] LustreError: 694331:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12367.334733] LustreError: 694331:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12367.982480] LustreError: 694358:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767327335 with bad export cookie 4279187927515632336 [12367.985960] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12367.986178] LustreError: 694358:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [12368.119722] Lustre: server umount lustre-MDT0001 complete [12373.296781] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [12374.302853] Key type lgssc unregistered [12374.431534] LNet: 699227:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12374.435177] LNetError: 699227:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [12374.442463] LNet: Removed LNI 192.168.203.152@tcp [12374.746290] Key type .llcrypt unregistered [12374.747897] Key type ._llcrypt unregistered [12386.702675] Key type ._llcrypt registered [12386.703729] Key type .llcrypt registered [12386.738304] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_hostid [12409.636503] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [12409.912593] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12409.929790] alg: No test for adler32 (adler32-zlib) [12410.775792] Lustre: Lustre: Build Version: 2.17.0_RC4_1_g3742a8b [12410.857549] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [12412.440169] Key type lgssc registered [12412.777822] Lustre: Echo OBD driver; http://www.lustre.org/ [12419.112654] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12422.831327] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12425.910215] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12429.197769] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12434.362205] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12434.382428] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12434.388365] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12435.467744] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12435.478963] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12435.514981] Lustre: lustre-MDT0000: new disk, initializing [12435.537281] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12435.543301] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12436.741147] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [12442.624131] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12442.650022] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12442.681099] Lustre: 708036:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12442.701911] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12442.703608] Lustre: Skipped 1 previous similar message [12442.737787] Lustre: lustre-MDT0001: new disk, initializing [12442.766782] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12442.777768] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12442.780632] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12443.931491] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [12446.850693] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12447.899354] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12451.914029] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12451.936266] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12452.038725] Lustre: lustre-OST0000: new disk, initializing [12452.040410] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12452.066274] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12452.789792] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12452.795520] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12452.831599] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12453.831073] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [12456.757405] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12457.980390] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12458.041364] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12459.259529] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12459.329261] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12463.077381] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12463.077953] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12463.079758] LustreError: Skipped 1 previous similar message [12463.080425] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12463.083023] Lustre: Skipped 1 previous similar message [12466.029922] LustreError: 711665:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [12466.067868] Lustre: server umount lustre-OST0000 complete [12468.193712] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12468.194426] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12468.197160] Lustre: Skipped 1 previous similar message [12468.200203] Lustre: Skipped 1 previous similar message [12473.313054] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12473.314853] Lustre: Skipped 2 previous similar messages [12473.599120] LustreError: 711914:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [12473.601250] LustreError: 711914:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [12473.627903] Lustre: server umount lustre-MDT0000 complete [12475.080057] LustreError: 708026:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767327442 with bad export cookie 940670862861609887 [12475.083352] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12475.084448] LustreError: 708026:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [12475.148803] LustreError: 712165:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [12475.150427] LustreError: 712165:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [12475.201451] Lustre: server umount lustre-MDT0001 complete [12477.869738] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [12478.425146] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 23:17:25 (1767327445) [12482.554183] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12482.752120] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12484.013919] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [12487.955544] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12488.098146] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12489.546366] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [12491.268735] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12492.428941] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12495.615585] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12497.373704] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [12499.015986] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12499.626196] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [12510.604027] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [12512.225725] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12512.227460] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12512.228837] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12512.236657] Lustre: Skipped 1 previous similar message [12517.346216] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12517.347810] Lustre: Skipped 2 previous similar messages [12517.999072] LustreError: 716060:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [12518.000901] LustreError: 716060:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [12518.032860] Lustre: server umount lustre-OST0000 complete [12522.465706] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12525.628748] LustreError: 716310:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [12525.630495] LustreError: 716310:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [12525.658539] Lustre: server umount lustre-MDT0000 complete [12527.150635] LustreError: 713200:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767327494 with bad export cookie 940670862861610762 [12527.153774] LustreError: 713200:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [12527.155228] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12527.274728] Lustre: server umount lustre-MDT0001 complete [12531.539379] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [12532.573833] Key type lgssc unregistered [12532.714511] LNet: 717235:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12532.718136] LNetError: 717235:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [12532.733434] LNet: Removed LNI 192.168.203.152@tcp [12533.065214] Key type .llcrypt unregistered [12533.066386] Key type ._llcrypt unregistered [12542.122288] Key type ._llcrypt registered [12542.123343] Key type .llcrypt registered [12542.160370] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [12542.258263] Key type .llcrypt unregistered [12542.259562] Key type ._llcrypt unregistered [12566.968656] Key type ._llcrypt registered [12566.969810] Key type .llcrypt registered [12567.009244] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [12567.316469] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12567.331409] alg: No test for adler32 (adler32-zlib) [12568.172586] Lustre: Lustre: Build Version: 2.17.0_RC4_1_g3742a8b [12568.245435] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [12569.824122] Key type lgssc registered [12570.148443] Lustre: Echo OBD driver; http://www.lustre.org/ [12572.575938] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12572.581045] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12573.739553] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12574.967093] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [12578.920166] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12579.023722] LustreError: 722720:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12579.026856] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12579.028730] LustreError: 722720:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12580.815438] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [12582.594580] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12583.139387] LustreError: 722696:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12583.143580] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [12583.145180] LustreError: 722696:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12586.183990] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12586.346354] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12587.657409] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [12596.705990] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12596.709199] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12596.709388] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12596.712906] Lustre: Skipped 1 previous similar message [12600.559081] LustreError: 725132:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [12600.594255] Lustre: server umount lustre-OST0000 complete [12604.898554] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [12604.900572] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12604.903545] Lustre: Skipped 1 previous similar message [12604.904970] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12606.945114] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12606.945663] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12608.127554] LustreError: 725382:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [12608.129222] LustreError: 725382:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [12608.155702] Lustre: server umount lustre-MDT0000 complete [12609.622157] LustreError: 722676:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767327576 with bad export cookie 13863974487731257873 [12609.625192] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12609.627688] LustreError: 722676:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [12609.687403] LustreError: 725632:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [12609.689156] LustreError: 725632:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [12609.732453] Lustre: server umount lustre-MDT0001 complete [12613.693783] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [12614.637935] Key type lgssc unregistered [12614.759396] LNet: 726308:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12614.761662] LNetError: 726308:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [12614.770300] LNet: Removed LNI 192.168.203.152@tcp [12615.044578] Key type .llcrypt unregistered [12615.045676] Key type ._llcrypt unregistered [12640.716332] Key type ._llcrypt registered [12640.717261] Key type .llcrypt registered [12640.756426] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [12641.290454] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12641.298087] alg: No test for adler32 (adler32-zlib) [12642.151484] Lustre: Lustre: Build Version: 2.17.0_RC4_1_g3742a8b [12642.240588] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [12643.824099] Key type lgssc registered [12644.199721] Lustre: Echo OBD driver; http://www.lustre.org/ [12646.682221] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12646.690355] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12662.112249] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [12667.232150] LustreError: 730729:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.152@tcp: failed processing log, type 4: rc = -110 [12693.856186] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12695.610957] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [12697.318571] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12700.723651] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12700.908189] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12700.932976] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [12702.205461] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [12706.029461] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12706.162218] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12707.500909] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [12709.224232] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12710.371774] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12713.444177] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12713.445209] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12713.448209] Lustre: Skipped 1 previous similar message [12713.452810] Lustre: Skipped 1 previous similar message [12718.563292] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12718.566208] Lustre: Skipped 1 previous similar message [12718.767081] LustreError: 733395:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [12718.805699] Lustre: server umount lustre-OST0000 complete [12721.632798] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12721.635664] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12721.638976] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12723.681075] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12723.681905] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12726.397240] LustreError: 733644:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [12726.399133] LustreError: 733644:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [12726.424381] Lustre: server umount lustre-MDT0000 complete [12727.952341] LustreError: 732751:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767327695 with bad export cookie 6101247273736540265 [12727.954984] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12727.955880] LustreError: 732751:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [12728.017603] LustreError: 733893:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [12728.019430] LustreError: 733893:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [12728.066583] Lustre: server umount lustre-MDT0001 complete [12732.320135] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [12733.291583] Key type lgssc unregistered [12733.427353] LNet: 734568:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12733.429573] LNetError: 734568:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [12733.438442] LNet: Removed LNI 192.168.203.152@tcp [12733.768393] Key type .llcrypt unregistered [12733.770224] Key type ._llcrypt unregistered [12767.350667] Key type ._llcrypt registered [12767.351964] Key type .llcrypt registered [12767.387913] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [12767.817134] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12767.824311] alg: No test for adler32 (adler32-zlib) [12768.673916] Lustre: Lustre: Build Version: 2.17.0_RC4_1_g3742a8b [12768.758127] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [12770.336219] Key type lgssc registered [12770.886480] Lustre: Echo OBD driver; http://www.lustre.org/ [12774.169288] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12774.178782] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12775.347966] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12776.698607] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [12781.805051] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12781.960568] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12783.306912] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [12785.368277] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12786.724181] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12790.045237] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12790.151591] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12791.896216] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [12793.572509] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12800.427274] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [12805.602435] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12805.603921] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12805.605695] Lustre: Skipped 2 previous similar messages [12805.608929] Lustre: Skipped 2 previous similar messages [12809.044080] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.52@tcp (stopping) [12810.722237] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12810.724066] Lustre: Skipped 1 previous similar message [12811.134498] LustreError: 741716:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [12811.172540] Lustre: server umount lustre-MDT0000 complete [12811.894745] LustreError: 741719:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.52@tcp arrived at 1767327779 with bad export cookie 12287095796283604728 [12812.530714] LustreError: 738998:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767327779 with bad export cookie 12287095796283604105 [12812.534193] LustreError: 738998:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [12812.534664] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12812.768862] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12812.771037] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12812.774909] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12815.841131] LustreError: 739931:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12815.845846] LustreError: 739931:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [12817.888991] LustreError: 739017:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12817.889324] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12817.893113] LustreError: 739017:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12817.894658] Lustre: Skipped 1 previous similar message [12818.608111] LustreError: 741920:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [12818.611117] LustreError: 741920:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [12818.645713] Lustre: server umount lustre-OST0000 complete [12821.343573] LustreError: 742369:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [12821.345476] LustreError: 742369:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [12821.401640] Lustre: server umount lustre-MDT0001 complete [12826.032132] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [12827.224376] Key type lgssc unregistered [12827.343530] LNet: 743044:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12827.346443] LNetError: 743044:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [12827.358484] LNet: Removed LNI 192.168.203.152@tcp [12827.819118] Key type .llcrypt unregistered [12827.821425] Key type ._llcrypt unregistered [12854.890382] Key type ._llcrypt registered [12854.891375] Key type .llcrypt registered [12854.937145] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [12855.332547] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12855.388137] alg: No test for adler32 (adler32-zlib) [12856.244495] Lustre: Lustre: Build Version: 2.17.0_RC4_1_g3742a8b [12856.330097] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [12857.912121] Key type lgssc registered [12858.245598] Lustre: Echo OBD driver; http://www.lustre.org/ [12860.775612] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12860.781040] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12861.950262] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12863.207687] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [12867.203639] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12867.326458] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12868.494395] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [12870.393648] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12871.783428] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12875.699950] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12875.810227] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12877.694883] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [12879.530682] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12887.081406] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [12892.133903] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12892.142430] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12892.146081] Lustre: Skipped 1 previous similar message [12895.062610] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.52@tcp (stopping) [12897.215749] LustreError: 750188:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [12897.242084] Lustre: server umount lustre-MDT0001 complete [12897.251261] LustreError: 747490:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12897.255872] LustreError: 747490:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [12902.369412] LustreError: 747490:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12902.369716] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12902.370740] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12902.370745] Lustre: Skipped 1 previous similar message [12902.371228] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12905.007084] LustreError: 750392:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [12905.008844] LustreError: 750392:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [12905.041327] Lustre: server umount lustre-OST0000 complete [12906.533688] LustreError: 750641:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [12906.535205] LustreError: 750641:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [12906.591787] Lustre: server umount lustre-MDT0000 complete [12912.007575] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [12913.001561] Key type lgssc unregistered [12913.129398] LNet: 751516:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12913.132266] LNetError: 751516:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [12913.141290] LNet: Removed LNI 192.168.203.152@tcp [12913.441147] Key type .llcrypt unregistered [12913.442562] Key type ._llcrypt unregistered [12927.940279] Key type ._llcrypt registered [12927.941282] Key type .llcrypt registered [12927.977038] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_hostid [12951.223447] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [12951.649759] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12951.658616] alg: No test for adler32 (adler32-zlib) [12952.507091] Lustre: Lustre: Build Version: 2.17.0_RC4_1_g3742a8b [12952.584232] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [12954.160114] Key type lgssc registered [12954.515650] Lustre: Echo OBD driver; http://www.lustre.org/ [12960.582491] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12964.713223] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12968.042705] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12971.141264] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12992.927521] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [13000.188776] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13004.931865] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13004.954594] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13004.961290] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13006.054771] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13006.068063] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13006.113510] Lustre: lustre-MDT0000: new disk, initializing [13006.140522] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13006.148332] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13007.462218] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [13011.201589] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13015.787551] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13015.810811] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13015.905945] Lustre: lustre-OST0000: new disk, initializing [13015.908657] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13015.911366] Lustre: Skipped 1 previous similar message [13015.935783] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13017.614447] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [13020.502881] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13022.719299] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [13022.723600] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [13022.731581] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [13029.873988] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13033.949810] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13033.971152] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13034.010860] Lustre: 764381:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13034.036180] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13034.071464] Lustre: lustre-MDT0001: new disk, initializing [13034.112630] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13034.125191] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [13034.127971] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [13035.408386] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [13038.770570] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [13038.835797] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [13040.024179] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [13042.138414] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 2 sec [13044.056448] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.52@tcp (stopping) [13046.242325] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13046.243214] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13046.243826] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13046.246951] Lustre: Skipped 1 previous similar message [13049.167238] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.52@tcp (stopping) [13049.169523] Lustre: Skipped 1 previous similar message [13049.980537] LustreError: 768704:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [13050.002409] Lustre: server umount lustre-MDT0001 complete [13056.483582] LustreError: 764388:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13056.490025] LustreError: 764388:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [13061.605224] LustreError: 765743:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13061.609356] LustreError: 765743:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [13063.648994] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13063.651020] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13063.654612] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13066.722701] LustreError: 764392:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13068.272129] LustreError: 768974:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [13068.273796] LustreError: 768974:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [13068.311863] Lustre: server umount lustre-OST0000 complete [13076.540972] LustreError: 769224:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [13076.542549] LustreError: 769224:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [13076.593052] Lustre: server umount lustre-MDT0000 complete [13081.893808] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [13082.794497] Key type lgssc unregistered [13082.907281] LNet: 770099:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13082.909541] LNetError: 770099:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [13082.915283] LNet: Removed LNI 192.168.203.152@tcp [13083.170203] Key type .llcrypt unregistered [13083.171153] Key type ._llcrypt unregistered [13100.900945] Key type ._llcrypt registered [13100.901847] Key type .llcrypt registered [13100.934448] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [13101.016279] Key type .llcrypt unregistered [13101.017374] Key type ._llcrypt unregistered [13123.282304] Key type ._llcrypt registered [13123.283374] Key type .llcrypt registered [13123.322309] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [13123.718166] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13123.767098] alg: No test for adler32 (adler32-zlib) [13124.619336] Lustre: Lustre: Build Version: 2.17.0_RC4_1_g3742a8b [13124.696619] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [13126.280182] Key type lgssc registered [13126.630579] Lustre: Echo OBD driver; http://www.lustre.org/ [13131.605547] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13134.893315] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13137.083021] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13140.247611] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13140.264410] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13140.269334] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13141.346672] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13141.356459] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13141.390926] Lustre: lustre-MDT0000: new disk, initializing [13141.412913] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13141.418596] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13142.584745] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [13147.576558] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13147.600703] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13147.630720] Lustre: 778317:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13147.650915] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13147.652519] Lustre: Skipped 1 previous similar message [13147.685977] Lustre: lustre-MDT0001: new disk, initializing [13147.715148] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13147.725794] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13147.728574] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13148.886362] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [13151.772079] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13152.896331] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13156.004798] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13156.027821] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13156.113875] Lustre: lustre-OST0000: new disk, initializing [13156.115671] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13156.139617] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13157.808283] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [13158.709017] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13158.712805] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13158.741994] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13160.735533] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13164.002737] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13164.003757] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13164.006235] Lustre: Skipped 2 previous similar messages [13164.007736] Lustre: Skipped 1 previous similar message [13169.121516] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13169.123082] Lustre: Skipped 3 previous similar messages [13169.344681] LustreError: 781191:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [13169.366267] Lustre: server umount lustre-MDT0000 complete [13170.711110] LustreError: 780457:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767328137 with bad export cookie 18057123330038988031 [13170.714088] LustreError: 780457:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [13170.714766] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13170.775885] LustreError: 781441:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [13170.777736] LustreError: 781441:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [13170.823130] Lustre: server umount lustre-MDT0001 complete [13182.130817] LustreError: 781641:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [13182.132635] LustreError: 781641:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [13182.169736] Lustre: server umount lustre-OST0000 complete [13185.382318] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 23:29:12 (1767328152) [13186.014320] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13190.004753] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13190.174219] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13191.393394] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [13195.088690] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13195.198187] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13196.444515] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [13198.090540] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13199.304253] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13202.625383] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13202.647969] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13202.708547] Lustre: Found index 0 for lustre-OST0000, updating log [13204.624415] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [13206.469941] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13215.082465] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [13220.325208] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13220.325589] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13220.326503] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13220.328181] Lustre: Skipped 1 previous similar message [13224.303116] LustreError: 785515:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [13224.305117] LustreError: 785515:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [13224.336927] Lustre: server umount lustre-OST0000 complete [13225.280827] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13226.466258] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13226.467330] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13226.473635] Lustre: Skipped 1 previous similar message [13226.475150] Lustre: Skipped 3 previous similar messages [13232.383150] LustreError: 785817:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [13232.386210] LustreError: 785817:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [13232.417526] Lustre: server umount lustre-MDT0000 complete [13232.610548] LustreError: 785309:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13232.615829] LustreError: 785309:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [13233.930206] LustreError: 784851:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767328201 with bad export cookie 18057123330038989410 [13233.935687] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13234.050809] Lustre: server umount lustre-MDT0001 complete [13236.768765] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 23:30:03 (1767328203) [13237.375523] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13237.747779] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13238.125592] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13238.440483] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13238.747225] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13239.055316] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13239.427417] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13239.748086] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13240.062296] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13240.388333] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13240.757693] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13241.085111] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13241.396575] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13241.726719] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13242.041321] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13242.423356] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13242.731288] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13243.055073] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13243.371434] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13243.686304] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13243.994894] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13244.317838] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13244.619021] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13244.944485] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13245.246366] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13245.644270] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13245.956464] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13246.267021] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13246.596352] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13247.032307] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13247.355369] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13247.667238] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13247.994519] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13248.338048] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13248.663035] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13249.006432] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13249.343101] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13249.712703] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13250.057146] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13250.370205] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13250.673317] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13250.985697] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13251.299262] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13251.609796] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13251.927401] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13252.246328] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13252.548843] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13252.941655] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13253.266257] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13253.581459] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13253.903170] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13258.417782] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13258.583329] LustreError: 789832:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13258.607102] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13258.608924] Lustre: Skipped 1 previous similar message [13259.873079] LNet: 1 local NIs in recovery (showing 1): 192.168.203.152@tcp [13259.940866] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [13263.842064] LustreError: 789832:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13263.893239] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13265.240057] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [13266.952079] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13268.220892] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13271.605741] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13271.642095] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13271.739920] Lustre: Found index 0 for lustre-OST0000, updating log [13273.680218] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [13275.472804] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13280.228717] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13280.228802] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13280.232584] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13280.235766] Lustre: Skipped 3 previous similar messages [13282.286162] LustreError: 792521:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [13282.288387] LustreError: 792521:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [13282.327098] Lustre: server umount lustre-OST0000 complete [13284.833182] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13284.836893] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13288.929384] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13288.931565] Lustre: Skipped 8 previous similar messages [13290.127922] Lustre: server umount lustre-MDT0000 complete [13290.977450] LustreError: 789831:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13290.984750] LustreError: 789831:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [13291.615095] LustreError: 791892:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767328258 with bad export cookie 18057123330038990481 [13291.617962] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13291.618307] LustreError: 791892:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 8 previous similar messages [13291.729274] Lustre: server umount lustre-MDT0001 complete [13292.653674] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13304.275031] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_hostid [13322.336277] LNet: 1 local NIs in recovery (showing 1): 192.168.203.152@tcp [13323.914211] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [13330.435151] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13334.056130] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13337.213354] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13340.558763] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13342.521582] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 23:31:49 (1767328309) [13349.069450] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [13350.089626] Key type lgssc unregistered [13350.213339] LNet: 802045:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13350.216323] LNetError: 802045:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [13464.935580] LNet: Removed LNI 192.168.203.152@tcp [13465.215576] Key type .llcrypt unregistered [13465.216627] Key type ._llcrypt unregistered [13488.077527] Key type ._llcrypt registered [13488.078787] Key type .llcrypt registered [13488.118529] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [13488.656639] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13488.670592] alg: No test for adler32 (adler32-zlib) [13489.521973] Lustre: Lustre: Build Version: 2.17.0_RC4_1_g3742a8b [13489.605264] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [13491.184143] Key type lgssc registered [13491.516215] Lustre: Echo OBD driver; http://www.lustre.org/ [13492.840204] LNet: Added LNI 192.168.203.152@tcp100 [8/256/0/180] [13493.119939] LNet: Added LNI 192.168.203.152@tcp101 [8/256/0/180] [13493.389723] LNet: Added LNI 192.168.203.152@tcp102 [8/256/0/180] [13493.657995] LNet: Added LNI 192.168.203.152@tcp103 [8/256/0/180] [13493.933125] LNet: Added LNI 192.168.203.152@tcp104 [8/256/0/180] [13494.204279] LNet: Added LNI 192.168.203.152@tcp105 [8/256/0/180] [13494.472809] LNet: Added LNI 192.168.203.152@tcp106 [8/256/0/180] [13494.734119] LNet: Added LNI 192.168.203.152@tcp107 [8/256/0/180] [13495.002790] LNet: Added LNI 192.168.203.152@tcp108 [8/256/0/180] [13495.266216] LNet: Added LNI 192.168.203.152@tcp109 [8/256/0/180] [13495.529682] LNet: Added LNI 192.168.203.152@tcp110 [8/256/0/180] [13495.801210] LNet: Added LNI 192.168.203.152@tcp111 [8/256/0/180] [13496.061109] LNet: Added LNI 192.168.203.152@tcp112 [8/256/0/180] [13496.319809] LNet: Added LNI 192.168.203.152@tcp113 [8/256/0/180] [13496.579561] LNet: Added LNI 192.168.203.152@tcp114 [8/256/0/180] [13496.848668] LNet: Added LNI 192.168.203.152@tcp115 [8/256/0/180] [13497.123864] LNet: Added LNI 192.168.203.152@tcp116 [8/256/0/180] [13497.392782] LNet: Added LNI 192.168.203.152@tcp117 [8/256/0/180] [13497.661229] LNet: Added LNI 192.168.203.152@tcp118 [8/256/0/180] [13497.942597] LNet: Added LNI 192.168.203.152@tcp119 [8/256/0/180] [13498.215442] LNet: Added LNI 192.168.203.152@tcp120 [8/256/0/180] [13498.498756] LNet: Added LNI 192.168.203.152@tcp121 [8/256/0/180] [13498.775321] LNet: Added LNI 192.168.203.152@tcp122 [8/256/0/180] [13499.047214] LNet: Added LNI 192.168.203.152@tcp123 [8/256/0/180] [13499.327901] LNet: Added LNI 192.168.203.152@tcp124 [8/256/0/180] [13499.604079] LNet: Added LNI 192.168.203.152@tcp125 [8/256/0/180] [13499.892854] LNet: Added LNI 192.168.203.152@tcp126 [8/256/0/180] [13500.173078] LNet: Added LNI 192.168.203.152@tcp127 [8/256/0/180] [13500.473462] LNet: Added LNI 192.168.203.152@tcp128 [8/256/0/180] [13500.764425] LNet: Added LNI 192.168.203.152@tcp129 [8/256/0/180] [13501.050603] LNet: Added LNI 192.168.203.152@tcp130 [8/256/0/180] [13501.328821] LNet: Added LNI 192.168.203.152@tcp131 [8/256/0/180] [13501.605094] LNet: Added LNI 192.168.203.152@tcp132 [8/256/0/180] [13501.887372] LNet: Added LNI 192.168.203.152@tcp133 [8/256/0/180] [13502.161239] LNet: Added LNI 192.168.203.152@tcp134 [8/256/0/180] [13502.424575] LNet: Added LNI 192.168.203.152@tcp135 [8/256/0/180] [13510.929481] Lustre: DEBUG MARKER: conf-sanity test_73c: @@@@@@ FAIL: unable to start ost1 [13518.859717] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [13519.745525] Key type lgssc unregistered [13519.888550] LNet: 810309:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13519.892846] LNetError: 810309:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [13519.894808] LNet: Removed LNI 192.168.203.152@tcp [13519.895869] LNet: Removed LNI 192.168.203.152@tcp100 [13519.896905] LNet: Removed LNI 192.168.203.152@tcp101 [13519.897939] LNet: Removed LNI 192.168.203.152@tcp102 [13519.899051] LNet: Removed LNI 192.168.203.152@tcp103 [13519.900184] LNet: Removed LNI 192.168.203.152@tcp104 [13519.901259] LNet: Removed LNI 192.168.203.152@tcp105 [13519.902342] LNet: Removed LNI 192.168.203.152@tcp106 [13519.903411] LNet: Removed LNI 192.168.203.152@tcp107 [13519.904458] LNet: Removed LNI 192.168.203.152@tcp108 [13519.905561] LNet: Removed LNI 192.168.203.152@tcp109 [13519.906598] LNet: Removed LNI 192.168.203.152@tcp110 [13519.907630] LNet: Removed LNI 192.168.203.152@tcp111 [13519.908692] LNet: Removed LNI 192.168.203.152@tcp112 [13519.909823] LNet: Removed LNI 192.168.203.152@tcp113 [13519.910849] LNet: Removed LNI 192.168.203.152@tcp114 [13519.911935] LNet: Removed LNI 192.168.203.152@tcp115 [13519.912967] LNet: Removed LNI 192.168.203.152@tcp116 [13519.914167] LNet: Removed LNI 192.168.203.152@tcp117 [13519.915334] LNet: Removed LNI 192.168.203.152@tcp118 [13519.916447] LNet: Removed LNI 192.168.203.152@tcp119 [13519.917572] LNet: Removed LNI 192.168.203.152@tcp120 [13519.918636] LNet: Removed LNI 192.168.203.152@tcp121 [13519.919846] LNet: Removed LNI 192.168.203.152@tcp122 [13519.920961] LNet: Removed LNI 192.168.203.152@tcp123 [13519.922046] LNet: Removed LNI 192.168.203.152@tcp124 [13519.923116] LNet: Removed LNI 192.168.203.152@tcp125 [13519.924236] LNet: Removed LNI 192.168.203.152@tcp126 [13519.925337] LNet: Removed LNI 192.168.203.152@tcp127 [13519.926406] LNet: Removed LNI 192.168.203.152@tcp128 [13519.927457] LNet: Removed LNI 192.168.203.152@tcp129 [13519.928587] LNet: Removed LNI 192.168.203.152@tcp130 [13519.929682] LNet: Removed LNI 192.168.203.152@tcp131 [13519.930784] LNet: Removed LNI 192.168.203.152@tcp132 [13519.931909] LNet: Removed LNI 192.168.203.152@tcp133 [13519.933146] LNet: Removed LNI 192.168.203.152@tcp134 [13519.938492] LNet: Removed LNI 192.168.203.152@tcp135 [13520.167457] Key type .llcrypt unregistered [13520.168364] Key type ._llcrypt unregistered [13529.848938] Key type ._llcrypt registered [13529.850103] Key type .llcrypt registered [13529.882252] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_hostid [13552.096857] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [13552.480285] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13552.491704] alg: No test for adler32 (adler32-zlib) [13553.331766] Lustre: Lustre: Build Version: 2.17.0_RC4_1_g3742a8b [13553.406496] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [13554.984105] Key type lgssc registered [13555.289527] Lustre: Echo OBD driver; http://www.lustre.org/ [13561.080290] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13564.638921] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13567.598302] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13570.685265] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13572.863639] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 23:35:39 (1767328539) [13580.014677] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [13580.914521] Key type lgssc unregistered [13581.024275] LNet: 819212:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13581.026341] LNetError: 819212:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [13581.037290] LNet: Removed LNI 192.168.203.152@tcp [13581.296278] Key type .llcrypt unregistered [13581.297236] Key type ._llcrypt unregistered [13604.670361] Key type ._llcrypt registered [13604.671900] Key type .llcrypt registered [13604.705850] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [13605.008177] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13605.053043] alg: No test for adler32 (adler32-zlib) [13605.896477] Lustre: Lustre: Build Version: 2.17.0_RC4_1_g3742a8b [13605.971921] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [13607.544076] Key type lgssc registered [13607.844178] Lustre: Echo OBD driver; http://www.lustre.org/ [13617.867740] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing lnet_if_list [13619.290705] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [13619.301284] LNet: Added LNI 192.168.203.152@tcp10 [8/256/0/180] [13621.161117] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [13621.173352] LNet: Added LNI 192.168.203.152@tcp11 [8/256/0/180] [13622.422430] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [13622.432707] LNet: Added LNI 192.168.203.152@tcp12 [8/256/0/180] [13623.609410] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [13623.618215] LNet: Added LNI 192.168.203.152@tcp13 [8/256/0/180] [13625.039384] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [13625.055769] LNet: Added LNI 192.168.203.152@tcp14 [8/256/0/180] [13626.718690] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [13626.728618] LNet: Added LNI 192.168.203.152@tcp15 [8/256/0/180] [13628.097289] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [13628.110599] LNet: Added LNI 192.168.203.152@tcp16 [8/256/0/180] [13629.690297] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [13629.702947] LNet: Added LNI 192.168.203.152@tcp17 [8/256/0/180] [13631.321819] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [13631.332205] LNet: Added LNI 192.168.203.152@tcp18 [8/256/0/180] [13632.707492] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [13632.716702] LNet: Added LNI 192.168.203.152@tcp19 [8/256/0/180] [13634.146408] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [13634.155783] LNet: Added LNI 192.168.203.152@tcp20 [8/256/0/180] [13635.665573] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [13635.676971] LNet: Added LNI 192.168.203.152@tcp21 [8/256/0/180] [13637.195074] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [13637.208524] LNet: Added LNI 192.168.203.152@tcp22 [8/256/0/180] [13638.778116] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [13638.787964] LNet: Added LNI 192.168.203.152@tcp23 [8/256/0/180] [13640.444198] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [13640.453199] LNet: Added LNI 192.168.203.152@tcp24 [8/256/0/180] [13642.105089] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [13642.114796] LNet: Added LNI 192.168.203.152@tcp25 [8/256/0/180] [13643.791848] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [13643.801447] LNet: Added LNI 192.168.203.152@tcp26 [8/256/0/180] [13645.487273] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [13645.496808] LNet: Added LNI 192.168.203.152@tcp27 [8/256/0/180] [13647.283140] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [13647.295200] LNet: Added LNI 192.168.203.152@tcp28 [8/256/0/180] [13649.148252] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [13649.157946] LNet: Added LNI 192.168.203.152@tcp29 [8/256/0/180] [13651.029410] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [13651.040772] LNet: Added LNI 192.168.203.152@tcp30 [8/256/0/180] [13652.943628] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [13652.962873] LNet: Added LNI 192.168.203.152@tcp31 [8/256/0/180] [13654.902233] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [13654.911907] LNet: Added LNI 192.168.203.152@tcp32 [8/256/0/180] [13656.907019] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [13656.916947] LNet: Added LNI 192.168.203.152@tcp33 [8/256/0/180] [13658.805372] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [13658.817412] LNet: Added LNI 192.168.203.152@tcp34 [8/256/0/180] [13660.807943] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [13660.817642] LNet: Added LNI 192.168.203.152@tcp35 [8/256/0/180] [13662.759944] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [13662.769496] LNet: Added LNI 192.168.203.152@tcp36 [8/256/0/180] [13664.686529] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [13664.697778] LNet: Added LNI 192.168.203.152@tcp37 [8/256/0/180] [13666.804713] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [13666.816693] LNet: Added LNI 192.168.203.152@tcp38 [8/256/0/180] [13668.820113] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [13668.829440] LNet: Added LNI 192.168.203.152@tcp39 [8/256/0/180] [13670.810124] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [13670.819709] LNet: Added LNI 192.168.203.152@tcp40 [8/256/0/180] [13672.866628] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [13672.875417] LNet: Added LNI 192.168.203.152@tcp41 [8/256/0/180] [13674.871325] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [13674.880724] LNet: Added LNI 192.168.203.152@tcp42 [8/256/0/180] [13676.955123] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [13676.964830] LNet: Added LNI 192.168.203.152@tcp43 [8/256/0/180] [13679.100302] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [13679.108854] LNet: Added LNI 192.168.203.152@tcp44 [8/256/0/180] [13681.702652] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [13681.711439] LNet: Added LNI 192.168.203.152@tcp45 [8/256/0/180] [13683.839206] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [13683.847748] LNet: Added LNI 192.168.203.152@tcp46 [8/256/0/180] [13686.040696] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [13686.049489] LNet: Added LNI 192.168.203.152@tcp47 [8/256/0/180] [13688.205492] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [13688.215885] LNet: Added LNI 192.168.203.152@tcp48 [8/256/0/180] [13690.481481] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [13690.490690] LNet: Added LNI 192.168.203.152@tcp49 [8/256/0/180] [13692.814047] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [13692.822820] LNet: Added LNI 192.168.203.152@tcp50 [8/256/0/180] [13858.767135] Lustre: DEBUG MARKER: conf-sanity test_73e: @@@@@@ FAIL: Mount fails on 192.168.203.152@tcp50 [13866.417501] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [13867.376741] Key type lgssc unregistered [13867.531494] LNet: 847234:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13867.534766] LNetError: 847234:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [13867.536939] LNet: Removed LNI 192.168.203.152@tcp [13867.538079] LNet: Removed LNI 192.168.203.152@tcp10 [13867.539271] LNet: Removed LNI 192.168.203.152@tcp11 [13867.540485] LNet: Removed LNI 192.168.203.152@tcp12 [13867.541640] LNet: Removed LNI 192.168.203.152@tcp13 [13867.542840] LNet: Removed LNI 192.168.203.152@tcp14 [13867.544023] LNet: Removed LNI 192.168.203.152@tcp15 [13867.545207] LNet: Removed LNI 192.168.203.152@tcp16 [13867.546344] LNet: Removed LNI 192.168.203.152@tcp17 [13867.547599] LNet: Removed LNI 192.168.203.152@tcp18 [13867.548756] LNet: Removed LNI 192.168.203.152@tcp19 [13867.549937] LNet: Removed LNI 192.168.203.152@tcp20 [13867.551102] LNet: Removed LNI 192.168.203.152@tcp21 [13867.552300] LNet: Removed LNI 192.168.203.152@tcp22 [13867.553626] LNet: Removed LNI 192.168.203.152@tcp23 [13867.554799] LNet: Removed LNI 192.168.203.152@tcp24 [13867.555974] LNet: Removed LNI 192.168.203.152@tcp25 [13867.557145] LNet: Removed LNI 192.168.203.152@tcp26 [13867.558312] LNet: Removed LNI 192.168.203.152@tcp27 [13867.559532] LNet: Removed LNI 192.168.203.152@tcp28 [13867.560699] LNet: Removed LNI 192.168.203.152@tcp29 [13867.561807] LNet: Removed LNI 192.168.203.152@tcp30 [13867.562957] LNet: Removed LNI 192.168.203.152@tcp31 [13867.564145] LNet: Removed LNI 192.168.203.152@tcp32 [13867.565391] LNet: Removed LNI 192.168.203.152@tcp33 [13867.566636] LNet: Removed LNI 192.168.203.152@tcp34 [13867.567800] LNet: Removed LNI 192.168.203.152@tcp35 [13867.568990] LNet: Removed LNI 192.168.203.152@tcp36 [13867.570160] LNet: Removed LNI 192.168.203.152@tcp37 [13867.571306] LNet: Removed LNI 192.168.203.152@tcp38 [13867.572605] LNet: Removed LNI 192.168.203.152@tcp39 [13867.573833] LNet: Removed LNI 192.168.203.152@tcp40 [13867.575030] LNet: Removed LNI 192.168.203.152@tcp41 [13867.576253] LNet: Removed LNI 192.168.203.152@tcp42 [13867.577467] LNet: Removed LNI 192.168.203.152@tcp43 [13867.578657] LNet: Removed LNI 192.168.203.152@tcp44 [13867.579794] LNet: Removed LNI 192.168.203.152@tcp45 [13867.580952] LNet: Removed LNI 192.168.203.152@tcp46 [13867.582112] LNet: Removed LNI 192.168.203.152@tcp47 [13867.583300] LNet: Removed LNI 192.168.203.152@tcp48 [13867.584539] LNet: Removed LNI 192.168.203.152@tcp49 [13867.589799] LNet: Removed LNI 192.168.203.152@tcp50 [13867.842357] Key type .llcrypt unregistered [13867.843410] Key type ._llcrypt unregistered [13892.176717] Key type ._llcrypt registered [13892.177680] Key type .llcrypt registered [13892.210404] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [13892.549227] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13892.586198] alg: No test for adler32 (adler32-zlib) [13893.421941] Lustre: Lustre: Build Version: 2.17.0_RC4_1_g3742a8b [13893.495886] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [13895.072110] Key type lgssc registered [13895.391327] Lustre: Echo OBD driver; http://www.lustre.org/ [13898.433254] Lustre: DEBUG MARKER: conf-sanity test_74: @@@@@@ FAIL: MDT start failed [13900.065040] Lustre: DEBUG MARKER: == conf-sanity test 75: The order of --index should be irrelevant ========================================================== 23:41:07 (1767328867) [13920.697560] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [13924.297908] Lustre: DEBUG MARKER: conf-sanity test_75: @@@@@@ FAIL: add mds1 failed for default params [13933.741962] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_hostid [13952.778459] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [13959.017156] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13962.776942] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13965.691693] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13968.740702] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13969.240575] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 13444 sec ========= 23:42:16 (1767328936) [13969.729306] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 23:42:16 (1767328936) === [13970.720226] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 23:42:17 (1767328937) === [13980.352145] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [13981.262714] Key type lgssc unregistered [13981.384338] LNet: 864779:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13981.386273] LNetError: 864779:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [13981.393277] LNet: Removed LNI 192.168.203.152@tcp [13981.627477] Key type .llcrypt unregistered [13981.628830] Key type ._llcrypt unregistered