[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 3.0.0 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.3-2.fc40 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 662266238 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f53f0-0x000f53ff] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5200 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1D87 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1C23 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001BE3 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1C97 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1D27 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1D5F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1c23-0xbffe1c96] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1c22] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1c97-0xbffe1d26] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1d27-0xbffe1d5e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1d5f-0xbffe1d86] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001014] APIC: Switch to symmetric I/O mode setup [ 0.002578] x2apic enabled [ 0.003010] Switched APIC routing to physical x2apic. [ 0.004016] kvm-guest: setup PV IPIs [ 0.007357] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008028] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009017] pid_max: default: 32768 minimum: 301 [ 0.010138] LSM: Security Framework initializing [ 0.011066] Yama: becoming mindful. [ 0.012000] SELinux: Initializing. [ 0.012082] *** VALIDATE selinux *** [ 0.021753] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026738] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027144] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028147] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029116] *** VALIDATE tmpfs *** [ 0.031472] *** VALIDATE proc *** [ 0.032203] *** VALIDATE cgroup *** [ 0.033009] *** VALIDATE cgroup2 *** [ 0.034247] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035130] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036006] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037028] Spectre V2 : User space: Vulnerable [ 0.038022] Speculative Store Bypass: Vulnerable [ 0.041593] debug: unmapping init [mem 0xffffffff99659000-0xffffffff99660fff] [ 0.044175] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045689] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046020] ... version: 2 [ 0.047009] ... bit width: 48 [ 0.048010] ... generic registers: 4 [ 0.049009] ... value mask: 0000ffffffffffff [ 0.050010] ... max period: 00007fffffffffff [ 0.051011] ... fixed-purpose events: 3 [ 0.052008] ... event mask: 000000070000000f [ 0.053274] rcu: Hierarchical SRCU implementation. [ 0.055327] smp: Bringing up secondary CPUs ... [ 0.056648] x86: Booting SMP configuration: [ 0.057020] .... node #0, CPUs: #1 #2 #3 [ 0.064149] smp: Brought up 1 node, 4 CPUs [ 0.066023] smpboot: Max logical packages: 1 [ 0.067013] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.108394] node 0 deferred pages initialised in 39ms [ 0.112218] devtmpfs: initialized [ 0.113250] x86/mm: Memory block size: 128MB [ 0.115619] gcov: version magic: 0x41383552 [ 0.117088] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.118193] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.119411] pinctrl core: initialized pinctrl subsystem [ 0.120269] [ 0.120676] ************************************************************* [ 0.121011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.122015] ** ** [ 0.123010] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.124012] ** ** [ 0.125010] ** This means that this kernel is built to expose internal ** [ 0.126011] ** IOMMU data structures, which may compromise security on ** [ 0.127013] ** your system. ** [ 0.128013] ** ** [ 0.129014] ** If you see this message and you are not debugging the ** [ 0.130014] ** kernel, report this immediately to your vendor! ** [ 0.131010] ** ** [ 0.132011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.133012] ************************************************************* [ 0.134985] NET: Registered protocol family 16 [ 0.135450] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.136061] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.137054] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.138777] cpuidle: using governor menu [ 0.140892] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.143478] PCI: Using configuration type 1 for base access [ 0.146138] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.158276] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.159020] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.162000] cryptd: max_cpu_qlen set to 1000 [ 0.167201] ACPI: Added _OSI(Module Device) [ 0.168009] ACPI: Added _OSI(Processor Device) [ 0.169009] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.170010] ACPI: Added _OSI(Processor Aggregator Device) [ 0.173602] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.177589] ACPI: Interpreter enabled [ 0.179041] ACPI: PM: (supports S0 S3 S4 S5) [ 0.180008] ACPI: Using IOAPIC for interrupt routing [ 0.182105] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.186473] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.211214] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.215034] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.223022] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.230080] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.238596] acpiphp: Slot [2] registered [ 0.241250] acpiphp: Slot [5] registered [ 0.245182] acpiphp: Slot [6] registered [ 0.248146] acpiphp: Slot [7] registered [ 0.250117] acpiphp: Slot [8] registered [ 0.252137] acpiphp: Slot [9] registered [ 0.254137] acpiphp: Slot [10] registered [ 0.255240] acpiphp: Slot [3] registered [ 0.257239] acpiphp: Slot [4] registered [ 0.258109] acpiphp: Slot [11] registered [ 0.260132] acpiphp: Slot [12] registered [ 0.262119] acpiphp: Slot [13] registered [ 0.263154] acpiphp: Slot [14] registered [ 0.265099] acpiphp: Slot [15] registered [ 0.267208] acpiphp: Slot [16] registered [ 0.269083] acpiphp: Slot [17] registered [ 0.271135] acpiphp: Slot [18] registered [ 0.273085] acpiphp: Slot [19] registered [ 0.276106] acpiphp: Slot [20] registered [ 0.278092] acpiphp: Slot [21] registered [ 0.279075] acpiphp: Slot [22] registered [ 0.280112] acpiphp: Slot [23] registered [ 0.282090] acpiphp: Slot [24] registered [ 0.283110] acpiphp: Slot [25] registered [ 0.285111] acpiphp: Slot [26] registered [ 0.288201] acpiphp: Slot [27] registered [ 0.290129] acpiphp: Slot [28] registered [ 0.292123] acpiphp: Slot [29] registered [ 0.294151] acpiphp: Slot [30] registered [ 0.297121] acpiphp: Slot [31] registered [ 0.299094] PCI host bridge to bus 0000:00 [ 0.302025] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.306154] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.312000] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.318027] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.324025] pci_bus 0000:00: root bus resource [mem 0x380000000000-0x38007fffffff window] [ 0.329024] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.332282] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.336986] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.346000] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.360863] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.365555] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.368015] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.371015] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.379034] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.383608] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.387312] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.391040] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.394807] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.399012] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.419015] pci 0000:00:02.0: reg 0x20: [mem 0x380000000000-0x380000003fff 64bit pref] [ 0.424016] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.429623] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.437017] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.450017] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.467016] pci 0000:00:05.0: reg 0x20: [mem 0x380000004000-0x380000007fff 64bit pref] [ 0.484047] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.493018] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.502000] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.522015] pci 0000:00:06.0: reg 0x20: [mem 0x380000008000-0x38000000bfff 64bit pref] [ 0.532000] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.579016] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.588015] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.606016] pci 0000:00:07.0: reg 0x20: [mem 0x38000000c000-0x38000000ffff 64bit pref] [ 0.624846] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.631013] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.636015] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.657023] pci 0000:00:08.0: reg 0x20: [mem 0x380000010000-0x380000013fff 64bit pref] [ 0.674184] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.685016] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.694015] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.718016] pci 0000:00:09.0: reg 0x20: [mem 0x380000014000-0x380000017fff 64bit pref] [ 0.728000] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.739024] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.753013] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.763018] pci 0000:00:0a.0: reg 0x20: [mem 0x380000018000-0x38000001bfff 64bit pref] [ 0.774000] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.775381] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.777346] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.780391] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.782279] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.789008] iommu: Default domain type: Passthrough [ 0.790592] SCSI subsystem initialized [ 0.792152] ACPI: bus type USB registered [ 0.794097] usbcore: registered new interface driver usbfs [ 0.796079] usbcore: registered new interface driver hub [ 0.798092] usbcore: registered new device driver usb [ 0.800192] pps_core: LinuxPPS API ver. 1 registered [ 0.802011] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.805053] PTP clock support registered [ 0.808041] EDAC MC: Ver: 3.0.0 [ 0.810150] PCI: Using ACPI for IRQ routing [ 0.811906] NetLabel: Initializing [ 0.813014] NetLabel: domain hash size = 128 [ 0.815009] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.817094] NetLabel: unlabeled traffic allowed by default [ 0.820111] vgaarb: loaded [ 0.824369] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.825050] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.835000] clocksource: Switched to clocksource kvm-clock [ 0.960576] VFS: Disk quotas dquot_6.6.0 [ 0.962206] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.964546] *** VALIDATE ramfs *** [ 0.965818] *** VALIDATE hugetlbfs *** [ 0.968176] pnp: PnP ACPI init [ 0.970568] pnp: PnP ACPI: found 6 devices [ 0.988217] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.991512] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.993732] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.995939] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.998403] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.000751] pci_bus 0000:00: resource 8 [mem 0x380000000000-0x38007fffffff window] [ 1.003589] NET: Registered protocol family 2 [ 1.005889] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.011204] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.016738] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.022337] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.025915] TCP: Hash tables configured (established 65536 bind 65536) [ 1.028352] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.031550] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.034677] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.037121] NET: Registered protocol family 1 [ 1.039921] RPC: Registered named UNIX socket transport module. [ 1.042723] RPC: Registered udp transport module. [ 1.044600] RPC: Registered tcp transport module. [ 1.045916] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.047575] NET: Registered protocol family 44 [ 1.049705] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.052181] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.053939] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.055730] PCI: CLS 0 bytes, default 64 [ 1.057778] Unpacking initramfs... [ 3.064117] debug: unmapping init [mem 0xffff90c07cc54000-0xffff90c07ffbffff] [ 3.068756] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 3.070806] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 3.073464] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.576956] Initialise system trusted keyrings [ 3.578462] Key type blacklist registered [ 3.580709] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.589293] zbud: loaded [ 3.592461] *** VALIDATE nfs *** [ 3.593674] *** VALIDATE nfs4 *** [ 3.595137] pstore: using deflate compression [ 3.598651] Platform Keyring initialized [ 3.745265] NET: Registered protocol family 38 [ 3.747434] Key type asymmetric registered [ 3.749258] Asymmetric key parser 'x509' registered [ 3.751345] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.755238] io scheduler mq-deadline registered [ 3.756790] io scheduler kyber registered [ 3.758373] io scheduler bfq registered [ 3.760808] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.765366] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.768490] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.771574] ACPI: Power Button [PWRF] [ 3.874171] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.992913] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 4.240957] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 4.750652] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 5.177704] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 5.223605] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 5.268194] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 5.275256] Non-volatile memory driver v1.3 [ 5.277364] Linux agpgart interface v0.103 [ 5.317522] virtio_blk virtio1: [vda] 133928 512-byte logical blocks (68.6 MB/65.4 MiB) [ 5.325217] vda: detected capacity change from 0 to 68571136 [ 5.391667] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 5.394322] vdb: detected capacity change from 0 to 1073741824 [ 5.428861] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 5.434305] vdc: detected capacity change from 0 to 2621440000 [ 5.464602] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 5.471304] vdd: detected capacity change from 0 to 2621440000 [ 5.502859] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 5.505577] vde: detected capacity change from 0 to 4294967296 [ 5.532077] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 5.542479] vdf: detected capacity change from 0 to 4294967296 [ 5.557029] libphy: Fixed MDIO Bus: probed [ 5.572134] usbcore: registered new interface driver usbserial_generic [ 5.574480] usbserial: USB Serial support registered for generic [ 5.578626] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 5.587542] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 5.589421] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 5.599069] mousedev: PS/2 mouse device common for all mice [ 5.604692] rtc_cmos 00:05: RTC can wake from S4 [ 5.608464] rtc_cmos 00:05: registered as rtc0 [ 5.610011] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 5.616381] intel_pstate: CPU model not supported [ 5.622686] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 5.629399] hid: raw HID events driver (C) Jiri Kosina [ 5.629557] usbcore: registered new interface driver usbhid [ 5.629560] usbhid: USB HID core driver [ 5.629656] drop_monitor: Initializing network drop monitor service [ 5.629776] Initializing XFRM netlink socket [ 5.630165] NET: Registered protocol family 10 [ 5.637572] Segment Routing with IPv6 [ 5.637609] NET: Registered protocol family 17 [ 5.647504] mpls_gso: MPLS GSO support [ 5.663665] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 5.699440] RAS: Correctable Errors collector initialized. [ 5.701162] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 5.707141] AVX version of gcm_enc/dec engaged. [ 5.721245] AES CTR mode by8 optimization enabled [ 5.915255] sched_clock: Marking stable (5915236014, 0)->(7050540825, -1135304811) [ 5.923785] registered taskstats version 1 [ 5.929327] Loading compiled-in X.509 certificates [ 5.931185] zswap: loaded using pool lzo/zbud [ 6.000557] Key type big_key registered [ 6.034533] Key type encrypted registered [ 6.035988] ima: No TPM chip found, activating TPM-bypass! [ 6.040304] ima: Allocated hash algorithm: sha1 [ 6.043855] ima: No architecture policies found [ 6.047481] evm: Initialising EVM extended attributes: [ 6.053418] evm: security.selinux [ 6.054519] evm: security.ima [ 6.055577] evm: security.capability [ 6.056796] evm: HMAC attrs: 0x1 [ 6.062626] rtc_cmos 00:05: setting system clock to 2025-12-03 03:40:04 UTC (1764733204) [ 6.073950] debug: unmapping init [mem 0xffffffff9a603000-0xffffffff9a7fffff] [ 6.087353] debug: unmapping init [mem 0xffffffff99382000-0xffffffff99658fff] [ 6.095203] Write protecting the kernel read-only data: 28672k [ 6.102321] debug: unmapping init [mem 0xffffffff97a03000-0xffffffff97bfffff] [ 6.105439] debug: unmapping init [mem 0xffffffff98314000-0xffffffff983fffff] [ 6.170661] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 6.188260] systemd[1]: Detected virtualization kvm. [ 6.190473] systemd[1]: Detected architecture x86-64. [ 6.192320] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 6.234449] systemd[1]: No hostname configured. [ 6.236481] systemd[1]: Set hostname to . [ 6.239104] random: systemd: uninitialized urandom read (16 bytes read) [ 6.245302] systemd[1]: Initializing machine ID from random generator. [ 6.442765] random: ln: uninitialized urandom read (6 bytes read) [ 6.844612] random: systemd: uninitialized urandom read (16 bytes read) [ 6.851688] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 6.857617] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 6.862247] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket. Starting Create Volatile Files and Directories... Starting Create list of required st…ce nodes for the current kernel... Starting Journal Service... [ OK ] Reached target Sockets. Starting Setup Virtual Console... [ OK ] Reached target Initrd Root Device. Starting Apply Kernel Variables... [ OK ] Reached target Swap. [ OK ] Started Memstrack Anylazing Service. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 8.714790] device-mapper: uevent: version 1.0.3 [ 8.898075] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] [ 9.869523] random: fast init done Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ 10.740592] virtio_net virtio0 ens2: renamed from eth0 [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 10.923931] scsi host0: ata_piix [ 10.946714] scsi host1: ata_piix [ 10.950552] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 10.962798] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 16.565315] random: crng init done [ 16.566956] random: 7 urandom warning(s) missed due to ratelimiting [ 17.569772] dracut-initqueue[582]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 19.218664] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 21.800986] printk: systemd: 26 output lines suppressed due to ratelimiting [ 22.497338] SELinux: Disabled at runtime. [ 22.567362] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 22.595685] systemd[1]: Detected virtualization kvm. [ 22.600466] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 24.340630] systemd[1]: initrd-switch-root.service: Succeeded. [ 24.345593] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 24.364781] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 24.382635] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 24.398385] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 24.416708] systemd[1]: Starting Journal Service... Starting Journal Service... [ 24.438892] systemd[1]: Created slice system-getty.slice. [ OK ] Created slice system-getty.slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Listening on initctl Compatibility Named Pipe. Mounting Kernel Debug File System... [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice User and Session Slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. Mounting POSIX Message Queue File System... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Reached target RPC Port Mapper. [ 24.788096] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Remount Root and Kernel File Systems... Mounting Huge Pages File System... [ OK ] Listening on Process Core Dump Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target Slices. [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 26.054880] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 27.467093] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 27.596447] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 28.422561] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 28.549466] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (8s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit) [*** ] A start job is running for Configur…-only root support (9s / no limit)[ 33.790680] Key type dns_resolver registered [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…only root support (10s / no limit)[ 34.881518] NFS: Registering the id_resolver key type [ 34.883598] Key type id_resolver registered [ 34.917706] Key type id_legacy registered [ ***] A start job is running for Configur…only root support (10s / no limit) [ **] A start job is running for Configur…only root support (11s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ 36.668143] hrtimer: interrupt took 2403908 ns [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started D-Bus System Message Bus. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Network Manager... [ OK ] Started irqbalance daemon. Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... [ OK ] Started GSSAPI Proxy Daemon. Starting Hostname Service... [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Login Service. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg105-server login: [ 90.180325] libcfs: loading out-of-tree module taints kernel. [ 90.222797] Key type ._llcrypt registered [ 90.224197] Key type .llcrypt registered [ 90.417625] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [ 108.328312] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 109.491079] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 109.506517] alg: No test for adler32 (adler32-zlib) [ 110.739537] Lustre: Lustre: Build Version: 2.17.0_RC1_3_g509e2bd [ 111.212875] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 112.895217] Key type lgssc registered [ 114.352708] Lustre: Echo OBD driver; http://www.lustre.org/ [ 128.583610] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 129.767555] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 136.785066] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 142.647982] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 148.857163] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 160.985805] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 171.159518] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 171.209628] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 171.233456] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 172.523178] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 172.563990] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 172.638729] Lustre: lustre-MDT0000: new disk, initializing [ 172.713851] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 172.731426] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 175.419925] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 186.046462] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 186.110464] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 186.167240] Lustre: 6508:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 186.194508] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 186.196360] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 186.197414] Lustre: Skipped 1 previous similar message [ 186.260346] Lustre: lustre-MDT0001: new disk, initializing [ 186.319140] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 186.338502] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 186.346705] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 189.617051] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 193.531359] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 201.433564] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 201.515947] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 201.683127] Lustre: lustre-OST0000: new disk, initializing [ 201.699640] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 201.766864] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 206.166969] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 207.371652] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 207.381222] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 207.465316] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 222.895662] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 222.957252] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 223.021332] Lustre: lustre-OST0001: new disk, initializing [ 223.023617] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 223.085472] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 229.461452] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 229.474992] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 229.611548] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 232.166564] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 250.092480] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 258.109372] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 269.899838] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing check_logdir /tmp/testlogs/ [ 274.254980] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing yml_node [ 279.811259] Lustre: DEBUG MARKER: Client: 2.17.0.RC1 [ 282.493875] Lustre: DEBUG MARKER: MDS: 2.17.0.RC1 [ 285.162568] Lustre: DEBUG MARKER: OSS: 2.17.0.RC1 [ 286.607917] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Dec 2 22:44:44 EST 2025 [ 303.256438] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 304.463588] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 319.455901] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 319.460476] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 319.477676] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 322.020784] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 322.025085] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 322.054658] Lustre: Skipped 3 previous similar messages [ 327.136044] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 327.147284] Lustre: Skipped 3 previous similar messages [ 332.258328] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 333.281430] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 333.404230] LustreError: 12211:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 333.477663] Lustre: server umount lustre-MDT0000 complete [ 336.734900] LustreError: 6498:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764733535 with bad export cookie 3767520696175829360 [ 336.735793] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 336.760773] LustreError: 6498:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 337.382027] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 337.404559] Lustre: Skipped 3 previous similar messages [ 341.415540] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 341.427803] Lustre: Skipped 6 previous similar messages [ 346.593713] LustreError: 6515:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 346.609263] LustreError: 6515:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 347.618601] LustreError: 10286:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 347.652356] LustreError: 10286:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 351.199138] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 351.320173] LustreError: 12416:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 351.323904] LustreError: 12416:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 351.405289] Lustre: server umount lustre-MDT0001 complete [ 354.875121] LustreError: 12619:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 354.885672] LustreError: 12619:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 354.943281] Lustre: server umount lustre-OST0000 complete [ 358.584259] LustreError: 12821:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 358.588666] LustreError: 12821:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 358.729473] Lustre: server umount lustre-OST0001 complete [ 364.403747] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [ 374.201128] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 384.427314] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 391.452264] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 397.523585] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 404.659278] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 411.560234] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 411.665096] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 411.906926] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 411.937514] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 412.009762] Lustre: lustre-MDT0000: new disk, initializing [ 412.166902] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 412.208985] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 416.160803] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 426.953581] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 427.020778] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 427.111268] Lustre: 15004:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 427.180077] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 427.182942] Lustre: Skipped 1 previous similar message [ 427.249177] Lustre: lustre-MDT0001: new disk, initializing [ 427.330409] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 427.367746] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 427.379477] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 431.212794] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 438.181266] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 441.878085] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 448.537870] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 448.614599] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 448.839824] Lustre: lustre-OST0000: new disk, initializing [ 448.843264] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 448.966043] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 450.663108] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 450.684021] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 450.817757] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 454.741673] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 462.594637] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 467.059944] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 467.363125] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 472.579737] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 472.819952] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 476.646218] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 476.650205] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 476.663491] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 476.689838] Lustre: Skipped 1 previous similar message [ 476.695708] Lustre: Skipped 4 previous similar messages [ 481.347180] LustreError: 17943:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 481.357078] LustreError: 17943:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 481.491279] Lustre: server umount lustre-OST0000 complete [ 486.881543] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 486.903235] Lustre: Skipped 1 previous similar message [ 490.904277] LustreError: 18146:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 490.908068] LustreError: 18146:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 490.970954] Lustre: server umount lustre-MDT0000 complete [ 492.002775] LustreError: 15887:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 492.013948] LustreError: 15887:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 494.420207] LustreError: 14998:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764733692 with bad export cookie 3767520696175830886 [ 494.433783] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 494.814182] Lustre: server umount lustre-MDT0001 complete [ 498.874464] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 22:48:16 (1764733696) [ 500.873578] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 502.419102] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 22:48:19 (1764733699) [ 508.976587] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 509.536859] LustreError: 19017:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 509.622893] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 513.656819] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 515.041895] LustreError: 19017:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 522.382333] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 526.223363] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 530.543509] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 534.016901] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 540.880667] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 546.020789] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 550.894387] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 563.687959] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 563.699288] LustreError: Skipped 1 previous similar message [ 563.711115] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 563.736932] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 563.748044] Lustre: Skipped 3 previous similar messages [ 565.732881] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 569.088176] LustreError: 21276:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 569.123795] LustreError: 21276:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 569.217920] Lustre: server umount lustre-OST0000 complete [ 573.923807] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 586.207296] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 586.353679] Lustre: server umount lustre-MDT0000 complete [ 588.853524] LustreError: 18997:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764733787 with bad export cookie 3767520696175831775 [ 588.857062] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 588.866568] LustreError: 18997:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 589.127249] Lustre: server umount lustre-MDT0001 complete [ 596.593712] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 599.045460] Key type lgssc unregistered [ 599.341848] LNet: 22215:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 599.352610] LNetError: 22215:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 599.380363] LNet: Removed LNI 192.168.201.105@tcp [ 600.114350] Key type .llcrypt unregistered [ 600.116600] Key type ._llcrypt unregistered [ 619.749914] Key type ._llcrypt registered [ 619.751700] Key type .llcrypt registered [ 619.870751] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 620.801818] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 620.880891] alg: No test for adler32 (adler32-zlib) [ 621.907618] Lustre: Lustre: Build Version: 2.17.0_RC1_3_g509e2bd [ 622.092467] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 623.751509] Key type lgssc registered [ 624.602115] Lustre: Echo OBD driver; http://www.lustre.org/ [ 631.356539] loop: module loaded [ 631.440781] loop0: detected capacity change from 0 to 204800000 [ 631.637370] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 631.644113] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 631.850203] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 631.894114] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 636.187724] loop0: detected capacity change from 0 to 204800000 [ 636.296392] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 636.305289] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 636.312455] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 636.413792] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 640.486211] loop0: detected capacity change from 0 to 204800000 [ 645.379566] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 645.446266] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 645.469465] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 646.720125] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 646.786462] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 646.890949] Lustre: lustre-MDT0000: new disk, initializing [ 646.975481] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 646.995757] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 650.707846] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 658.758178] loop1: detected capacity change from 0 to 204800000 [ 663.464824] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 663.532981] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 663.825947] Lustre: lustre-OST07c6: new disk, initializing [ 663.831074] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 663.834185] Lustre: Skipped 1 previous similar message [ 663.922837] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 668.444650] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 673.912711] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 673.920822] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 673.975521] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 678.314710] Lustre: 24369:0:(osd_internal.h:1459:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 678.331293] Lustre: 24369:0:(osd_handler.c:2096:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 678.336899] Lustre: 24369:0:(osd_handler.c:2103:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 678.354396] Lustre: 24369:0:(osd_handler.c:2113:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 678.368703] Lustre: 24369:0:(osd_handler.c:2120:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 678.378292] Lustre: 24369:0:(osd_handler.c:2127:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 695.798365] Lustre: 24370:0:(osd_internal.h:1459:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 695.810419] Lustre: 24370:0:(osd_handler.c:2096:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 695.818481] Lustre: 24370:0:(osd_handler.c:2103:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 695.823822] Lustre: 24370:0:(osd_handler.c:2113:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/4 [ 695.828775] Lustre: 24370:0:(osd_handler.c:2120:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 695.833884] Lustre: 24370:0:(osd_handler.c:2127:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 700.432437] Lustre: 24369:0:(osd_internal.h:1459:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 700.449714] Lustre: 24369:0:(osd_internal.h:1459:osd_trans_exec_op()) Skipped 1 previous similar message [ 700.467392] Lustre: 24369:0:(osd_handler.c:2096:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 700.472369] Lustre: 24369:0:(osd_handler.c:2096:osd_trans_dump_creds()) Skipped 1 previous similar message [ 700.476435] Lustre: 24369:0:(osd_handler.c:2103:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 700.481018] Lustre: 24369:0:(osd_handler.c:2103:osd_trans_dump_creds()) Skipped 1 previous similar message [ 700.485762] Lustre: 24369:0:(osd_handler.c:2113:osd_trans_dump_creds()) write: 2/23/0, punch: 0/0/0, quota 4/150/0 [ 700.490426] Lustre: 24369:0:(osd_handler.c:2113:osd_trans_dump_creds()) Skipped 1 previous similar message [ 700.494055] Lustre: 24369:0:(osd_handler.c:2120:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 700.503214] Lustre: 24369:0:(osd_handler.c:2120:osd_trans_dump_creds()) Skipped 1 previous similar message [ 700.510709] Lustre: 24369:0:(osd_handler.c:2127:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 700.518395] Lustre: 24369:0:(osd_handler.c:2127:osd_trans_dump_creds()) Skipped 1 previous similar message [ 703.699562] Lustre: 24371:0:(osd_internal.h:1459:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 703.708105] Lustre: 24371:0:(osd_internal.h:1459:osd_trans_exec_op()) Skipped 1 previous similar message [ 703.716565] Lustre: 24371:0:(osd_handler.c:2096:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 703.721324] Lustre: 24371:0:(osd_handler.c:2096:osd_trans_dump_creds()) Skipped 1 previous similar message [ 703.731081] Lustre: 24371:0:(osd_handler.c:2103:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 703.740748] Lustre: 24371:0:(osd_handler.c:2103:osd_trans_dump_creds()) Skipped 1 previous similar message [ 703.752965] Lustre: 24371:0:(osd_handler.c:2113:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/0 [ 703.773262] Lustre: 24371:0:(osd_handler.c:2113:osd_trans_dump_creds()) Skipped 1 previous similar message [ 703.780513] Lustre: 24371:0:(osd_handler.c:2120:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 703.793823] Lustre: 24371:0:(osd_handler.c:2120:osd_trans_dump_creds()) Skipped 1 previous similar message [ 703.797689] Lustre: 24371:0:(osd_handler.c:2127:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 703.809484] Lustre: 24371:0:(osd_handler.c:2127:osd_trans_dump_creds()) Skipped 1 previous similar message [ 708.109049] Lustre: 24371:0:(osd_internal.h:1459:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 708.123482] Lustre: 24371:0:(osd_internal.h:1459:osd_trans_exec_op()) Skipped 1 previous similar message [ 708.127539] Lustre: 24371:0:(osd_handler.c:2096:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 708.131082] Lustre: 24371:0:(osd_handler.c:2096:osd_trans_dump_creds()) Skipped 1 previous similar message [ 708.135853] Lustre: 24371:0:(osd_handler.c:2103:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 708.139974] Lustre: 24371:0:(osd_handler.c:2103:osd_trans_dump_creds()) Skipped 1 previous similar message [ 708.144075] Lustre: 24371:0:(osd_handler.c:2113:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/0 [ 708.149440] Lustre: 24371:0:(osd_handler.c:2113:osd_trans_dump_creds()) Skipped 1 previous similar message [ 708.153524] Lustre: 24371:0:(osd_handler.c:2120:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 708.158578] Lustre: 24371:0:(osd_handler.c:2120:osd_trans_dump_creds()) Skipped 1 previous similar message [ 708.162079] Lustre: 24371:0:(osd_handler.c:2127:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 708.169887] Lustre: 24371:0:(osd_handler.c:2127:osd_trans_dump_creds()) Skipped 1 previous similar message [ 718.852170] Lustre: 24369:0:(osd_internal.h:1459:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 718.883511] Lustre: 24369:0:(osd_internal.h:1459:osd_trans_exec_op()) Skipped 5 previous similar messages [ 718.903633] Lustre: 24369:0:(osd_handler.c:2096:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 718.912169] Lustre: 24369:0:(osd_handler.c:2096:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 718.923030] Lustre: 24369:0:(osd_handler.c:2103:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 718.929580] Lustre: 24369:0:(osd_handler.c:2103:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 718.940031] Lustre: 24369:0:(osd_handler.c:2113:osd_trans_dump_creds()) write: 2/23/0, punch: 0/0/0, quota 4/150/0 [ 718.944880] Lustre: 24369:0:(osd_handler.c:2113:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 718.949078] Lustre: 24369:0:(osd_handler.c:2120:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 718.952512] Lustre: 24369:0:(osd_handler.c:2120:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 718.956203] Lustre: 24369:0:(osd_handler.c:2127:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 718.959800] Lustre: 24369:0:(osd_handler.c:2127:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 765.921494] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 765.933732] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 771.043278] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 776.160554] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 779.231173] Lustre: lustre-OST07c6 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 779.273159] LustreError: 26558:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 779.350334] Lustre: server umount lustre-OST07c6 complete [ 786.139434] LustreError: 27060:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 786.141888] LustreError: 27060:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 786.266270] Lustre: server umount lustre-MDT0000 complete [ 795.883802] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 797.708206] Key type lgssc unregistered [ 797.913528] LNet: 27892:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 797.917314] LNetError: 27892:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 797.932630] LNet: Removed LNI 192.168.201.105@tcp [ 798.457479] Key type .llcrypt unregistered [ 798.459896] Key type ._llcrypt unregistered [ 810.992939] Key type ._llcrypt registered [ 810.994527] Key type .llcrypt registered [ 811.071840] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 811.590228] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 811.649667] alg: No test for adler32 (adler32-zlib) [ 812.634582] Lustre: Lustre: Build Version: 2.17.0_RC1_3_g509e2bd [ 812.773594] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 814.415211] Key type lgssc registered [ 815.247199] Lustre: Echo OBD driver; http://www.lustre.org/ [ 821.412862] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 821.423957] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 822.794641] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 825.534175] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 831.612814] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 831.866509] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 834.201695] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 837.607088] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 840.006902] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 844.558996] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 844.823904] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 847.927382] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 848.729471] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 852.417601] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 866.789255] LustreError: lustre-OST0000-osc-MDT0000: operation ost_setattr to node 0@lo failed: rc = -107 [ 866.792641] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 866.799313] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 867.298373] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 867.299768] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 867.311402] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 867.316509] LustreError: Skipped 1 previous similar message [ 867.322534] Lustre: Skipped 1 previous similar message [ 868.213240] LustreError: 31571:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 868.307903] Lustre: server umount lustre-OST0000 complete [ 872.421973] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 872.422791] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 872.433456] Lustre: Skipped 1 previous similar message [ 872.446062] Lustre: Skipped 1 previous similar message [ 877.207554] LustreError: 31771:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 877.213231] LustreError: 31771:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 877.266678] Lustre: server umount lustre-MDT0000 complete [ 877.536979] LustreError: 29835:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 877.548820] LustreError: 29835:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 879.485731] LustreError: 29105:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764734077 with bad export cookie 5394402756179762817 [ 879.490127] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 879.493110] LustreError: 29105:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 879.599607] LustreError: 31974:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 879.603070] LustreError: 31974:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 879.711975] Lustre: server umount lustre-MDT0001 complete [ 887.344232] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 889.012194] Key type lgssc unregistered [ 889.228180] LNet: 32505:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 889.234455] LNetError: 32505:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 889.247785] LNet: Removed LNI 192.168.201.105@tcp [ 889.905362] Key type .llcrypt unregistered [ 889.909447] Key type ._llcrypt unregistered [ 902.262661] Key type ._llcrypt registered [ 902.264864] Key type .llcrypt registered [ 902.339096] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 902.934912] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 902.968635] alg: No test for adler32 (adler32-zlib) [ 904.002373] Lustre: Lustre: Build Version: 2.17.0_RC1_3_g509e2bd [ 904.156626] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 905.791189] Key type lgssc registered [ 906.720923] Lustre: Echo OBD driver; http://www.lustre.org/ [ 912.684645] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 912.704275] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 914.042227] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 916.543769] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 921.607348] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 921.839617] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 924.201349] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 928.132270] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 930.812792] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 935.988593] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 936.296850] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 939.385719] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 940.360585] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 944.017320] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 957.926377] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 957.935379] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 957.942536] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 957.953487] Lustre: Skipped 1 previous similar message [ 962.747380] LustreError: 36003:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 963.007796] Lustre: server umount lustre-OST0000 complete [ 973.284137] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 973.291878] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 973.306904] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 973.327836] Lustre: Skipped 2 previous similar messages [ 975.511406] LustreError: 36206:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 975.544918] LustreError: 36206:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 975.870065] Lustre: server umount lustre-MDT0000 complete [ 978.400782] LustreError: 33736:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 978.423176] LustreError: 33736:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 979.433802] LustreError: 33736:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 979.476356] LustreError: 33736:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 980.037343] LustreError: 36207:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764734178 with bad export cookie 17114595192736666715 [ 980.049047] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 980.058292] LustreError: 36207:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 980.310785] LustreError: 36409:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 980.317417] LustreError: 36409:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 980.605136] Lustre: server umount lustre-MDT0001 complete [ 992.239984] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 995.243486] Key type lgssc unregistered [ 995.527857] LNet: 36941:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 995.544257] LNetError: 36941:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 995.560936] LNet: Removed LNI 192.168.201.105@tcp [ 996.379380] Key type .llcrypt unregistered [ 996.380757] Key type ._llcrypt unregistered [ 1018.410692] Key type ._llcrypt registered [ 1018.412259] Key type .llcrypt registered [ 1018.501521] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 1019.440722] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1019.673965] alg: No test for adler32 (adler32-zlib) [ 1020.796111] Lustre: Lustre: Build Version: 2.17.0_RC1_3_g509e2bd [ 1020.994278] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 1022.673366] Key type lgssc registered [ 1023.540109] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1031.322164] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1031.350684] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1032.779818] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1036.036480] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 1041.812371] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1042.061836] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1044.763858] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 1048.924744] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1051.738300] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1056.867040] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1057.122788] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1058.169570] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1062.202230] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 1066.537494] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1074.268284] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.5@tcp (stopping) [ 1075.033047] LustreError: 40583:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1075.123579] Lustre: server umount lustre-MDT0000 complete [ 1077.734305] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1077.737586] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1077.738740] LustreError: 39051:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1077.738750] LustreError: 39051:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1077.754437] Lustre: Skipped 2 previous similar messages [ 1078.166094] LustreError: 38318:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764734276 with bad export cookie 2123237868792969102 [ 1078.183233] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1079.361896] LustreError: 38334:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1079.381493] LustreError: 38334:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1079.386548] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.5@tcp (stopping) [ 1082.849896] LustreError: 38335:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1082.851155] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1082.893211] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1084.335982] LustreError: 40784:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1084.346342] LustreError: 40784:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1084.449900] Lustre: server umount lustre-MDT0001 complete [ 1104.788992] LustreError: 41007:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1104.791621] LustreError: 41007:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1104.900540] Lustre: server umount lustre-OST0000 complete [ 1119.006657] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 1121.346829] Key type lgssc unregistered [ 1121.632160] LNet: 41844:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1121.637264] LNetError: 41844:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1122.661544] LNet: Removed LNI 192.168.201.105@tcp [ 1123.382557] Key type .llcrypt unregistered [ 1123.385625] Key type ._llcrypt unregistered [ 1140.961225] Key type ._llcrypt registered [ 1140.963125] Key type .llcrypt registered [ 1141.048902] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 1141.620190] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1141.657315] alg: No test for adler32 (adler32-zlib) [ 1142.658663] Lustre: Lustre: Build Version: 2.17.0_RC1_3_g509e2bd [ 1142.857156] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 1144.511196] Key type lgssc registered [ 1145.406235] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1151.215785] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1151.233974] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1152.641214] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1155.802411] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 1158.116792] LustreError: 43219:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1158.137318] LustreError: 43219:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1162.956820] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1163.364744] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1166.899216] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 1170.874294] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1173.878662] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1179.462227] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1179.762339] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1184.323525] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 1188.601756] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1198.209359] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1205.846726] Lustre: lustre-OST0000: Not available for connect from 192.168.201.5@tcp (stopping) [ 1208.810851] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1208.813382] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1208.822874] LustreError: Skipped 1 previous similar message [ 1208.839102] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1208.847565] Lustre: Skipped 1 previous similar message [ 1210.951610] Lustre: lustre-OST0000: Not available for connect from 192.168.201.5@tcp (stopping) [ 1211.006982] LustreError: 45488:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1211.119250] Lustre: server umount lustre-OST0000 complete [ 1229.159500] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1229.163981] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1229.170973] Lustre: Skipped 2 previous similar messages [ 1229.177756] Lustre: Skipped 1 previous similar message [ 1233.291555] LustreError: 45862:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1233.294880] LustreError: 45862:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1233.380973] Lustre: server umount lustre-MDT0000 complete [ 1234.405368] LustreError: 43215:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1234.425867] LustreError: 43215:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1236.202096] LustreError: 43201:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764734434 with bad export cookie 12814589634456536625 [ 1236.208503] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1236.209679] LustreError: 43201:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1242.432155] LustreError: 46064:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1242.434847] LustreError: 46064:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1242.634768] Lustre: server umount lustre-MDT0001 complete [ 1250.820391] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 1253.103441] Key type lgssc unregistered [ 1253.309741] LNet: 46597:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1253.315369] LNetError: 46597:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1254.373652] LNet: Removed LNI 192.168.201.105@tcp [ 1255.129108] Key type .llcrypt unregistered [ 1255.130935] Key type ._llcrypt unregistered [ 1274.708885] Key type ._llcrypt registered [ 1274.710431] Key type .llcrypt registered [ 1274.802907] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 1275.594866] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1275.603582] alg: No test for adler32 (adler32-zlib) [ 1276.582783] Lustre: Lustre: Build Version: 2.17.0_RC1_3_g509e2bd [ 1276.781887] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 1278.463337] Key type lgssc registered [ 1279.436764] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1285.310227] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1285.334545] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1286.767253] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1289.694968] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 1292.268691] LustreError: 47985:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1292.310589] LustreError: 47985:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1297.376293] LustreError: 47985:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1298.076703] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1298.480259] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1302.353100] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 1308.535903] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1312.244847] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1318.664847] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1319.015272] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1323.816793] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 1329.277889] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1332.348679] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1338.747576] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1341.305283] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1352.909961] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2025-12-0223h02m30s [ 1354.269689] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1356.366315] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.5@tcp (stopping) [ 1357.444534] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.5@tcp (stopping) [ 1358.304901] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1358.313473] Lustre: Skipped 2 previous similar messages [ 1359.329367] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1359.332111] Lustre: Skipped 5 previous similar messages [ 1362.006726] LustreError: 50551:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1362.071232] Lustre: server umount lustre-MDT0000 complete [ 1362.499169] LustreError: 49459:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1364.449035] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1365.478401] LustreError: 47985:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1365.497756] LustreError: 47985:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1369.897456] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1369.993271] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1370.209613] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1370.300584] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1371.249730] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1373.557373] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 1376.938654] Lustre: DEBUG MARKER: Wait for df (36137) ... [ 1378.506620] Lustre: DEBUG MARKER: done [ 1381.862097] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1381.873829] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1381.876678] Lustre: Skipped 1 previous similar message [ 1386.976573] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1386.990583] Lustre: Skipped 2 previous similar messages [ 1387.638225] LustreError: 51607:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1387.648476] LustreError: 51607:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1387.740277] Lustre: server umount lustre-OST0000 complete [ 1392.102606] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1392.102973] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1396.388722] LustreError: 51809:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1396.397260] LustreError: 51809:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1396.466500] Lustre: server umount lustre-MDT0000 complete [ 1398.242179] LustreError: 47984:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1398.249853] LustreError: 47984:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1399.138633] LustreError: 47966:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764734597 with bad export cookie 13992145299283377676 [ 1399.146908] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1399.263421] LustreError: 52010:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1399.267547] LustreError: 52010:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1399.367251] Lustre: server umount lustre-MDT0001 complete [ 1406.883896] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 1408.651527] Key type lgssc unregistered [ 1408.855289] LNet: 52541:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1408.860607] LNetError: 52541:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1408.871157] LNet: Removed LNI 192.168.201.105@tcp [ 1409.365445] Key type .llcrypt unregistered [ 1409.368154] Key type ._llcrypt unregistered [ 1432.680610] Key type ._llcrypt registered [ 1432.682025] Key type .llcrypt registered [ 1432.771981] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [ 1443.195908] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 1443.801799] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1443.828904] alg: No test for adler32 (adler32-zlib) [ 1444.764660] Lustre: Lustre: Build Version: 2.17.0_RC1_3_g509e2bd [ 1444.909764] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 1446.544384] Key type lgssc registered [ 1447.281686] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1453.211613] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1457.628522] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1462.732094] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1467.711278] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1472.681987] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1472.712748] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1472.724560] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1473.934235] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1473.960558] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1474.021723] Lustre: lustre-MDT0000: new disk, initializing [ 1474.096562] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1474.115580] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1476.920983] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 1484.748655] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1484.797382] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1484.859882] Lustre: 56467:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1484.932704] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1484.935900] Lustre: Skipped 1 previous similar message [ 1484.994265] Lustre: lustre-MDT0001: new disk, initializing [ 1485.082394] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1485.118169] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1485.137213] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1488.389743] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 1494.219438] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1496.842813] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1502.260854] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1502.320965] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1502.518921] Lustre: lustre-OST0000: new disk, initializing [ 1502.521872] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1502.601934] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1506.728044] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 1512.492268] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1512.498795] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1512.635813] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1512.671661] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1515.809228] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1515.998481] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1518.608962] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1518.776808] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1523.169246] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1523.169428] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1523.181612] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1523.187817] LustreError: Skipped 1 previous similar message [ 1523.193223] Lustre: Skipped 1 previous similar message [ 1526.083158] LustreError: 59403:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1526.185922] Lustre: server umount lustre-OST0000 complete [ 1531.360516] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1531.366586] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1531.379349] Lustre: Skipped 1 previous similar message [ 1531.385586] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1533.413238] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1533.415282] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1538.528846] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1538.534928] Lustre: Skipped 2 previous similar messages [ 1542.625276] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1542.629975] Lustre: Skipped 1 previous similar message [ 1543.135239] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1543.197949] LustreError: 59606:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1543.201317] LustreError: 59606:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1543.272389] Lustre: server umount lustre-MDT0000 complete [ 1546.194315] LustreError: 56456:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764734744 with bad export cookie 8127037384605319082 [ 1546.197266] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1546.213969] LustreError: 56456:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1546.356782] LustreError: 59810:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1546.366374] LustreError: 59810:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1546.566287] Lustre: server umount lustre-MDT0001 complete [ 1552.898921] Lustre: DEBUG MARKER: == conf-sanity test 35b: Continue reconnection retries, if the active server is busy ========================================================== 23:05:50 (1764734750) [ 1557.345277] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1557.657433] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1560.071271] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 1563.105205] LustreError: 60479:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1563.123342] LustreError: 60479:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1566.586505] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1566.876330] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1569.965624] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 1574.392978] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1577.456866] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1582.197634] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1582.438269] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1586.719753] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 1590.838805] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1596.687351] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2025-12-0223h06m34s [ 1597.872795] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1602.556671] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1604.521633] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1604.576153] LustreError: 60474:0:(mdt_handler.c:2334:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1624.660051] Lustre: lustre-MDT0000: Client 2c52210d-5364-4889-b10b-c49d15fb41db (at 192.168.201.5@tcp) reconnecting [ 1644.663513] LustreError: 60474:0:(mdt_handler.c:2334:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1645.870390] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1647.248972] Lustre: DEBUG MARKER: done [ 1653.218427] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1653.220850] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1653.229583] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1653.237618] Lustre: Skipped 1 previous similar message [ 1656.504103] LustreError: 63483:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1656.520531] LustreError: 63483:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1656.594154] Lustre: server umount lustre-OST0000 complete [ 1663.457471] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1663.472160] Lustre: Skipped 1 previous similar message [ 1665.237329] LustreError: 63685:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1665.241033] LustreError: 63685:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1665.291073] Lustre: server umount lustre-MDT0000 complete [ 1667.595121] LustreError: 60461:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764734866 with bad export cookie 8127037384605319964 [ 1667.599445] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1667.602379] LustreError: 60461:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1667.789903] Lustre: server umount lustre-MDT0001 complete [ 1672.834523] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 1674.686147] Key type lgssc unregistered [ 1674.870797] LNet: 64421:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1674.875332] LNetError: 64421:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1674.884223] LNet: Removed LNI 192.168.201.105@tcp [ 1675.332834] Key type .llcrypt unregistered [ 1675.334198] Key type ._llcrypt unregistered [ 1693.105476] Key type ._llcrypt registered [ 1693.107214] Key type .llcrypt registered [ 1693.173520] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [ 1702.515853] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 1703.369204] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1703.380294] alg: No test for adler32 (adler32-zlib) [ 1704.352552] Lustre: Lustre: Build Version: 2.17.0_RC1_3_g509e2bd [ 1704.534532] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 1706.193371] Key type lgssc registered [ 1706.851372] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1712.203877] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1716.324120] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1720.072458] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1724.359765] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1728.820382] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1728.850308] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1728.860852] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1729.997368] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1730.020364] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1730.065814] Lustre: lustre-MDT0000: new disk, initializing [ 1730.133353] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1730.150842] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1732.579946] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 1738.924578] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1738.958242] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1739.020934] Lustre: 68346:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1739.061876] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1739.065231] Lustre: Skipped 1 previous similar message [ 1739.120603] Lustre: lustre-MDT0001: new disk, initializing [ 1739.187937] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1739.216436] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1739.222300] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1741.371360] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 1746.472799] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1748.706437] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1752.636401] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1752.680242] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1752.830402] Lustre: lustre-OST0000: new disk, initializing [ 1752.833849] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1752.880181] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1754.794294] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1754.801101] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1754.879365] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1756.073480] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 1760.873417] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1763.234793] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1763.381309] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1766.018741] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1766.170582] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1770.466197] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1770.467611] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1770.475434] LustreError: Skipped 1 previous similar message [ 1770.476206] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1770.481704] Lustre: Skipped 1 previous similar message [ 1770.491897] Lustre: Skipped 1 previous similar message [ 1773.504876] LustreError: 71285:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1773.566074] Lustre: server umount lustre-OST0000 complete [ 1780.192491] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1780.197407] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1780.204959] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1782.240426] LustreError: 71487:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1782.243532] LustreError: 71487:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1782.289498] Lustre: server umount lustre-MDT0000 complete [ 1784.984403] LustreError: 70091:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764734983 with bad export cookie 5331934175859020975 [ 1784.997193] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1785.120815] LustreError: 71688:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1785.124229] LustreError: 71688:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1785.220274] Lustre: server umount lustre-MDT0001 complete [ 1790.419058] Lustre: DEBUG MARKER: == conf-sanity test 36: df report consistency on OSTs with different block size ========================================================== 23:09:48 (1764734988) [ 1791.312651] Lustre: DEBUG MARKER: SKIP: conf-sanity test_36 remote OST [ 1792.422464] Lustre: DEBUG MARKER: == conf-sanity test 37: verify set tunables works for symlink device ========================================================== 23:09:50 (1764734990) [ 1799.278589] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 1801.547342] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1801.869733] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1802.512147] Lustre: Failing over lustre-MDT0000 [ 1802.515555] LustreError: 72795:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1802.522336] LustreError: 72795:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1802.527804] LustreError: 72795:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 0, retries 0, failed: rc = -5 [ 1802.617356] LustreError: 72846:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1802.619768] LustreError: 72846:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1802.719212] Lustre: server umount lustre-MDT0000 complete [ 1806.920219] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 23:10:04 (1764735004) [ 1811.216575] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1811.490713] LustreError: 73416:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1811.516033] LustreError: 73416:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1811.533545] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1813.602748] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 1816.547339] LustreError: 73416:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1818.544379] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1821.174357] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 1824.981892] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1827.295340] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1832.018689] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1832.284316] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1832.287580] Lustre: Skipped 1 previous similar message [ 1836.380766] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 1839.969675] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1847.237862] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1854.948635] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1854.948826] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1854.950633] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1854.950641] Lustre: Skipped 1 previous similar message [ 1854.964799] Lustre: Skipped 3 previous similar messages [ 1856.904658] LustreError: 75759:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1856.908214] LustreError: 75759:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1856.956370] Lustre: server umount lustre-MDT0000 complete [ 1859.054133] LustreError: 75760:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764735057 with bad export cookie 5331934175859022130 [ 1859.058589] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1859.068543] LustreError: 75760:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 1859.237427] Lustre: server umount lustre-MDT0001 complete [ 1861.120719] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1863.873717] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1869.223434] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1876.447149] Lustre: 66560:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1764735058/real 1764735058] req@000000009cdbd9b6 x1850458664662784/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1764735074 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1876.471784] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1880.479324] Lustre: 66562:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1764735062/real 1764735062] req@0000000069ed0c70 x1850458664662912/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1764735078 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1881.503165] Lustre: 66562:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1764735063/real 1764735063] req@00000000f1b1dc18 x1850458664663040/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1764735079 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1883.679929] LustreError: 66558:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000a973f93f x1850458664663936/t0(0) o250->MGC192.168.201.105@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1883.879809] LustreError: 75024:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1883.936318] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1883.982831] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1884.960420] Lustre: 66560:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1764735067/real 1764735067] req@000000003961e4f5 x1850458664663296/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1764735083 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1885.969458] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 1885.986308] LustreError: 76819:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1885.993530] LustreError: 76819:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1890.079177] Lustre: 66560:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1764735072/real 1764735072] req@0000000038c4bf31 x1850458664663424/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1764735088 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1890.555072] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1892.763870] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 1894.888197] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1895.677460] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1897.508410] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1901.024793] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1901.029886] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1901.040971] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1901.043975] Lustre: Skipped 2 previous similar messages [ 1906.444792] LustreError: 78220:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1906.450046] LustreError: 78220:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 1906.487161] Lustre: server umount lustre-MDT0000 complete [ 1908.193520] LustreError: 75024:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1908.201209] LustreError: 75024:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1908.237399] LustreError: 73395:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764735106 with bad export cookie 5331934175859024034 [ 1908.239703] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1908.242210] LustreError: 73395:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1908.413191] Lustre: server umount lustre-MDT0001 complete [ 1911.616647] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1916.728433] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1917.855594] LustreError: 79264:0:(import.c:337:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [ 1917.861065] LustreError: 79264:0:(import.c:361:ptlrpc_invalidate_import()) @@@ still on sending list req@0000000069ed0c70 x1850458664684160/t0(0) o250->MGC192.168.201.105@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1764735116 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1917.872934] LustreError: 79264:0:(import.c:371:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. [ 1918.431889] LustreError: 66558:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000083ab5c2a x1850458664684672/t0(0) o250->MGC192.168.201.105@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1918.591905] LustreError: 75024:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1918.634385] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1918.637923] Lustre: Skipped 1 previous similar message [ 1918.679461] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 1920.533892] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 1924.067075] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1924.074599] Lustre: Skipped 2 previous similar messages [ 1924.827639] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1927.083271] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 1928.992405] Lustre: 66562:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1764735111/real 1764735111] req@000000000053cb9a x1850458664684416/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1764735127 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1929.006479] Lustre: 66562:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1930.080435] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1930.219056] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1930.222764] Lustre: Skipped 1 previous similar message [ 1932.246318] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1933.285300] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1938.404161] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1938.407781] Lustre: Skipped 5 previous similar messages [ 1941.509820] LustreError: 80675:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1941.512483] LustreError: 80675:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 1941.557972] Lustre: server umount lustre-MDT0000 complete [ 1943.347347] LustreError: 73396:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764735141 with bad export cookie 5331934175859025406 [ 1943.349936] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1943.354689] LustreError: 73396:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1943.517970] Lustre: server umount lustre-MDT0001 complete [ 1946.781498] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1948.745660] Lustre: DEBUG MARKER: files compared the same [ 1960.071862] Lustre: server umount lustre-OST0000 complete [ 1967.628617] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 1969.018971] Key type lgssc unregistered [ 1969.163674] LNet: 82307:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1969.167592] LNetError: 82307:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1969.174868] LNet: Removed LNI 192.168.201.105@tcp [ 1969.567215] Key type .llcrypt unregistered [ 1969.569168] Key type ._llcrypt unregistered [ 1982.021824] Key type ._llcrypt registered [ 1982.023674] Key type .llcrypt registered [ 1982.088349] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 1982.600405] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1982.648408] alg: No test for adler32 (adler32-zlib) [ 1983.616426] Lustre: Lustre: Build Version: 2.17.0_RC1_3_g509e2bd [ 1983.739550] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 1985.343240] Key type lgssc registered [ 1985.935433] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1990.417687] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1990.432543] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1991.753387] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1993.834348] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug +malloc all [ 1998.030469] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1998.196659] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1999.992127] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug +malloc all [ 2002.663372] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2004.500762] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2008.776188] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2008.990386] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2012.048583] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug +malloc all [ 2015.114038] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2017.189820] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2032.607965] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2032.611262] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2032.617608] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2033.631895] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2033.632331] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2033.633301] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2035.380132] LustreError: 85950:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2035.424379] Lustre: server umount lustre-OST0000 complete [ 2038.753371] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2038.759094] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2038.762552] Lustre: Skipped 2 previous similar messages [ 2043.353873] LustreError: 86152:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2043.357302] LustreError: 86152:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2043.388218] Lustre: server umount lustre-MDT0000 complete [ 2043.873358] LustreError: 83679:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2043.885092] LustreError: 83679:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2045.283746] LustreError: 83665:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764735243 with bad export cookie 2752794764046610564 [ 2045.286517] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2045.289863] LustreError: 83665:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2045.385281] LustreError: 86363:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2045.387623] LustreError: 86363:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2045.460685] Lustre: server umount lustre-MDT0001 complete [ 2050.137956] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 2051.504308] Key type lgssc unregistered [ 2051.694189] LNet: 86896:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2051.698390] LNetError: 86896:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2052.714790] LNet: Removed LNI 192.168.201.105@tcp [ 2053.147183] Key type .llcrypt unregistered [ 2053.149078] Key type ._llcrypt unregistered [ 2065.037913] Key type ._llcrypt registered [ 2065.039536] Key type .llcrypt registered [ 2065.093244] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 2065.506189] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2065.574096] alg: No test for adler32 (adler32-zlib) [ 2066.476390] Lustre: Lustre: Build Version: 2.17.0_RC1_3_g509e2bd [ 2066.612193] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 2068.215311] Key type lgssc registered [ 2068.749351] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2072.543879] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2072.557451] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2088.031428] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2093.151237] LustreError: 88262:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.105@tcp: failed processing log, type 4: rc = -110 [ 2119.775217] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2122.363669] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2124.803410] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2128.810788] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2128.883733] Lustre: *** cfs_fail_loc=706, val=0*** [ 2129.092944] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2129.144366] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2131.746356] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2134.245619] LustreError: 89234:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2134.255565] LustreError: 89234:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2139.622303] LustreError: 88288:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2139.633826] LustreError: 88288:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2142.202350] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2142.625876] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2147.705398] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2154.940806] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2159.574953] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2163.169962] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2163.174978] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2163.216672] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2163.233018] Lustre: Skipped 1 previous similar message [ 2168.295647] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2168.310337] Lustre: Skipped 2 previous similar messages [ 2168.466147] LustreError: 90587:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 2168.607235] Lustre: server umount lustre-OST0000 complete [ 2173.410264] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2173.410356] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2173.413244] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2173.421889] Lustre: Skipped 1 previous similar message [ 2178.530210] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2178.533940] Lustre: Skipped 2 previous similar messages [ 2183.650116] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2183.654455] Lustre: Skipped 1 previous similar message [ 2186.208284] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2186.273298] LustreError: 90789:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 2186.276633] LustreError: 90789:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2186.337532] Lustre: server umount lustre-MDT0000 complete [ 2188.769975] LustreError: 89239:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2188.780731] LustreError: 89239:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2188.789771] LustreError: 88271:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764735387 with bad export cookie 2113322386769842907 [ 2188.791303] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2188.798384] LustreError: 88271:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2188.893425] LustreError: 90994:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2188.899074] LustreError: 90994:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2188.978614] Lustre: server umount lustre-MDT0001 complete [ 2196.599858] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 2198.765654] Key type lgssc unregistered [ 2198.975945] LNet: 91527:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2198.983966] LNetError: 91527:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2198.995946] LNet: Removed LNI 192.168.201.105@tcp [ 2199.607761] Key type .llcrypt unregistered [ 2199.611624] Key type ._llcrypt unregistered [ 2217.356303] Key type ._llcrypt registered [ 2217.358144] Key type .llcrypt registered [ 2217.473758] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 2218.440701] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2218.459751] alg: No test for adler32 (adler32-zlib) [ 2219.454942] Lustre: Lustre: Build Version: 2.17.0_RC1_3_g509e2bd [ 2219.661815] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 2221.327275] Key type lgssc registered [ 2222.220839] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2228.525582] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2228.543983] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2232.521199] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2238.173522] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2238.596140] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2241.756809] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2244.064294] LustreError: 93588:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2244.077477] LustreError: 93588:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2248.052402] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2248.160881] LustreError: 94302:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2248.176125] LustreError: 94302:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2248.340859] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2252.541235] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2253.793019] LustreError: 93592:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2253.805670] LustreError: 93592:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2258.738236] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2258.747113] Lustre: MGS: Client baff69ec-8304-4fb2-b81c-15ff7d9d2130 (at 0@lo) reconnecting [ 2258.918796] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2258.989416] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2261.875228] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2274.274311] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2274.276692] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2274.284076] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2274.288106] Lustre: Skipped 1 previous similar message [ 2279.283159] LustreError: 95552:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 2279.394237] Lustre: server umount lustre-OST0000 complete [ 2284.514997] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2284.515547] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2284.525045] Lustre: Skipped 1 previous similar message [ 2284.528408] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2284.532059] Lustre: Skipped 2 previous similar messages [ 2288.418922] LustreError: 95754:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2288.440724] LustreError: 95754:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2288.463050] Lustre: server umount lustre-MDT0000 complete [ 2289.632081] LustreError: 93591:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2289.665097] LustreError: 93591:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2291.846270] LustreError: 95958:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2291.854627] LustreError: 95958:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2291.998755] Lustre: server umount lustre-MDT0001 complete [ 2294.797964] LustreError: 96158:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2294.800759] LustreError: 96158:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2294.841437] Lustre: server umount MGS complete [ 2302.483975] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 23:18:20 (1764735500) [ 2322.391445] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [ 2329.114284] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 2335.351559] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2339.521709] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2343.981750] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2348.627497] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2353.026896] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2353.074320] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2355.778192] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2362.264627] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2362.301619] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2362.430432] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2362.472917] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2362.530384] Lustre: lustre-MDT0001: new disk, initializing [ 2362.587799] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2362.611374] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2362.619656] Lustre: 100673:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2362.625601] LustreError: 100673:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2365.466122] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2373.092271] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2373.156902] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2373.317853] Lustre: lustre-OST0000: new disk, initializing [ 2373.326604] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2373.375246] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2375.969930] Lustre: 101555:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@0000000064939d9d x1850459205702144/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1764735590 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2377.244734] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2382.303464] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2387.028927] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2387.043279] Lustre: MGS: Client f620e526-2c66-406d-baf0-143551655fbb (at 0@lo) reconnecting [ 2387.076328] Lustre: 99956:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2387.088902] LustreError: 99956:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x0 : rc = -11 [ 2387.109398] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2387.171678] Lustre: lustre-MDT0000: new disk, initializing [ 2387.320294] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2387.361468] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2387.447525] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2388.448844] Lustre: 102384:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@0000000030c6e6d5 x1850459205708288/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1764735602 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2390.236431] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2396.654724] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 2396.659548] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 2397.542727] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 2397.611511] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000401 [ 2417.121273] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2417.124297] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2417.138584] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2419.681127] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2419.685010] Lustre: Skipped 2 previous similar messages [ 2422.397821] LustreError: 102908:0:(obd_class.h:479:obd_check_dev()) Device 11 not setup [ 2422.406362] LustreError: 102908:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 2422.467886] Lustre: server umount lustre-OST0000 complete [ 2424.800773] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2424.801606] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2424.807958] LustreError: Skipped 1 previous similar message [ 2424.815930] Lustre: Skipped 1 previous similar message [ 2424.819185] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2424.821850] Lustre: Skipped 1 previous similar message [ 2430.672732] LustreError: 103111:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 2430.675908] LustreError: 103111:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2430.690532] Lustre: server umount lustre-MDT0000 complete [ 2438.872708] Lustre: server umount lustre-MDT0001 complete [ 2440.839979] Lustre: server umount MGS complete [ 2446.587710] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 23:20:44 (1764735644) [ 2454.523969] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 2455.758830] Key type lgssc unregistered [ 2455.899368] LNet: 104850:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2455.902972] LNetError: 104850:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2455.910896] LNet: Removed LNI 192.168.201.105@tcp [ 2456.262577] Key type .llcrypt unregistered [ 2456.264113] Key type ._llcrypt unregistered [ 2464.805691] Key type ._llcrypt registered [ 2464.807369] Key type .llcrypt registered [ 2464.864295] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 2465.299981] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2465.357146] alg: No test for adler32 (adler32-zlib) [ 2466.257371] Lustre: Lustre: Build Version: 2.17.0_RC1_3_g509e2bd [ 2466.396600] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 2468.015750] Key type lgssc registered [ 2468.588474] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2472.282739] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2472.291555] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2473.541533] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2475.385787] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2479.342273] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2479.530340] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2481.237416] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2483.695237] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2485.363697] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2488.492652] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2488.658939] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2491.130254] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2493.622725] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2497.903308] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:33) [ 2503.137797] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2503.139434] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2503.140264] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2503.144391] Lustre: Skipped 1 previous similar message [ 2508.227634] LustreError: 108277:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2508.289807] Lustre: server umount lustre-OST0000 complete [ 2510.304240] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2510.308586] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2510.318332] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2510.321911] Lustre: Skipped 1 previous similar message [ 2513.378125] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2513.379806] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2516.039440] LustreError: 108478:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2516.042071] LustreError: 108478:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2516.081409] Lustre: server umount lustre-MDT0000 complete [ 2517.827850] LustreError: 108190:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764735716 with bad export cookie 14427316022097906272 [ 2517.830084] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2517.833825] LustreError: 108190:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2517.922603] LustreError: 108680:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 2517.925915] LustreError: 108680:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2518.009909] Lustre: server umount lustre-MDT0001 complete [ 2522.474835] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 2523.787722] Key type lgssc unregistered [ 2523.937915] LNet: 109211:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2523.943457] LNetError: 109211:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2523.954635] LNet: Removed LNI 192.168.201.105@tcp [ 2524.329702] Key type .llcrypt unregistered [ 2524.331374] Key type ._llcrypt unregistered [ 2532.122422] Key type ._llcrypt registered [ 2532.124439] Key type .llcrypt registered [ 2532.180555] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 2532.595539] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2532.663936] alg: No test for adler32 (adler32-zlib) [ 2533.581952] Lustre: Lustre: Build Version: 2.17.0_RC1_3_g509e2bd [ 2533.692658] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 2535.303185] Key type lgssc registered [ 2535.819097] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2538.351574] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2538.351630] LustreError: 110252:0:(tgt_mount.c:2435:server_fill_super()) cfs_race id 716 sleeping [ 2538.354939] Lustre: Skipped 1 previous similar message [ 2538.360288] LustreError: 110253:0:(tgt_mount.c:2435:server_fill_super()) cfs_fail_race id 716 waking [ 2538.364104] LustreError: 110252:0:(tgt_mount.c:2435:server_fill_super()) cfs_fail_race id 716 awake: rc=4994 [ 2538.368872] LustreError: 110252:0:(tgt_mount.c:2447:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 2538.374660] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2538.374921] LustreError: 110252:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2539.610412] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2543.009836] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2543.162405] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2544.857746] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2547.084059] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing lsmod [ 2548.443126] LustreError: 111428:0:(tgt_mount.c:2435:server_fill_super()) cfs_race id 716 sleeping [ 2548.446679] LustreError: 111429:0:(tgt_mount.c:2435:server_fill_super()) cfs_fail_race id 716 waking [ 2548.450952] LustreError: 111428:0:(tgt_mount.c:2435:server_fill_super()) cfs_fail_race id 716 awake: rc=4996 [ 2548.453582] LustreError: 111428:0:(tgt_mount.c:2447:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 2548.458980] LustreError: 111428:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2548.470585] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2548.691456] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2549.750928] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:65) [ 2553.186486] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2553.189309] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2553.194840] Lustre: Skipped 1 previous similar message [ 2553.204757] Lustre: Skipped 2 previous similar messages [ 2556.235068] LustreError: 111619:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2556.289441] Lustre: server umount lustre-MDT0000 complete [ 2558.027843] LustreError: 110272:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764735756 with bad export cookie 4674358714283251765 [ 2558.031019] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2558.033660] LustreError: 110272:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2558.432298] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2558.433097] LustreError: 111581:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2558.438832] Lustre: Skipped 1 previous similar message [ 2558.444735] LustreError: 111581:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2558.447624] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2559.327234] LustreError: 110726:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000260c5938 x1850459534935552/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2559.340497] LustreError: 111823:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 2559.343654] LustreError: 111823:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2559.406797] Lustre: server umount lustre-MDT0001 complete [ 2561.116173] LustreError: 112023:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2561.118920] LustreError: 112023:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2561.176272] Lustre: server umount lustre-OST0000 complete [ 2565.323190] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2565.576749] LustreError: 112499:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2565.615177] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2567.566165] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2570.721211] LustreError: 112499:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2572.148324] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2572.373495] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2574.516676] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2577.291263] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2579.102827] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2582.820194] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2582.996109] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2585.969501] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2588.765764] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2591.207358] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:97) [ 2606.051578] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 2606.055195] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2606.062382] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2607.591392] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2608.628488] LustreError: 114769:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2608.632724] LustreError: 114769:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2608.682116] Lustre: server umount lustre-OST0000 complete [ 2612.704909] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2612.706559] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2612.714556] Lustre: Skipped 2 previous similar messages [ 2612.717211] Lustre: Skipped 2 previous similar messages [ 2616.595750] LustreError: 114970:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2616.598514] LustreError: 114970:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2616.658646] Lustre: server umount lustre-MDT0000 complete [ 2617.826851] LustreError: 113611:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2617.836134] LustreError: 113611:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2618.491191] LustreError: 112478:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764735816 with bad export cookie 4674358714283252675 [ 2618.494065] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2618.499556] LustreError: 112478:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2618.662461] Lustre: server umount lustre-MDT0001 complete [ 2623.763971] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 2625.192536] Key type lgssc unregistered [ 2625.348460] LNet: 115702:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2625.356409] LNetError: 115702:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2625.370106] LNet: Removed LNI 192.168.201.105@tcp [ 2625.789810] Key type .llcrypt unregistered [ 2625.791817] Key type ._llcrypt unregistered [ 2638.657725] Key type ._llcrypt registered [ 2638.659870] Key type .llcrypt registered [ 2638.720536] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 2639.175654] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2639.209745] alg: No test for adler32 (adler32-zlib) [ 2640.106591] Lustre: Lustre: Build Version: 2.17.0_RC1_3_g509e2bd [ 2640.210386] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 2641.807226] Key type lgssc registered [ 2642.383715] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2646.149663] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2646.165143] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2647.440537] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2649.481420] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2654.078403] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2654.286082] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2656.397822] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2659.381851] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2661.211790] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2665.091520] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2665.305913] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2668.245820] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2670.816485] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2672.493041] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:99 to 0x240000401:129) [ 2684.613613] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 2686.423491] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 2688.993693] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2688.993745] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2688.994539] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2689.012920] Lustre: Skipped 1 previous similar message [ 2694.114560] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2694.119048] Lustre: Skipped 1 previous similar message [ 2694.705103] LustreError: 119442:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2694.757090] Lustre: server umount lustre-OST0000 complete [ 2699.234023] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2699.235193] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2699.243990] Lustre: Skipped 1 previous similar message [ 2699.246869] Lustre: Skipped 1 previous similar message [ 2702.812404] LustreError: 119647:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2702.816941] LustreError: 119647:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2702.875366] Lustre: server umount lustre-MDT0000 complete [ 2704.353968] LustreError: 118112:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2704.361976] LustreError: 118112:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2704.851900] LustreError: 118697:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764735903 with bad export cookie 3290005920522261607 [ 2704.855569] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2704.859328] LustreError: 118697:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2704.955955] LustreError: 119848:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 2704.961367] LustreError: 119848:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2705.054352] Lustre: server umount lustre-MDT0001 complete [ 2710.233576] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 2711.640606] Key type lgssc unregistered [ 2711.811957] LNet: 120379:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2711.817055] LNetError: 120379:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2711.825447] LNet: Removed LNI 192.168.201.105@tcp [ 2712.273610] Key type .llcrypt unregistered [ 2712.275631] Key type ._llcrypt unregistered [ 2721.375221] Key type ._llcrypt registered [ 2721.376545] Key type .llcrypt registered [ 2721.441107] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 2721.969322] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2721.991922] alg: No test for adler32 (adler32-zlib) [ 2722.929725] Lustre: Lustre: Build Version: 2.17.0_RC1_3_g509e2bd [ 2723.080720] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 2724.711230] Key type lgssc registered [ 2725.270022] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2728.957368] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2728.969325] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2730.220940] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2732.016278] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2736.211699] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2736.428778] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2738.408840] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2740.955279] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2742.782518] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2746.148787] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2746.323625] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2749.001610] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2751.627752] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2755.561358] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:131 to 0x240000401:161) [ 2765.793609] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2765.794543] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2765.795525] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2765.798766] LustreError: Skipped 1 previous similar message [ 2765.817460] Lustre: Skipped 1 previous similar message [ 2767.285147] LustreError: 123825:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2767.369209] Lustre: server umount lustre-OST0000 complete [ 2770.913180] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2770.921343] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2770.924453] Lustre: Skipped 2 previous similar messages [ 2775.466044] LustreError: 124028:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2775.470020] LustreError: 124028:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2775.521558] Lustre: server umount lustre-MDT0000 complete [ 2776.031985] LustreError: 121560:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2776.040294] LustreError: 121560:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2777.636913] LustreError: 123826:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764735976 with bad export cookie 15884255898394885729 [ 2777.638524] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2777.642153] LustreError: 123826:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2777.738741] LustreError: 124230:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 2777.741246] LustreError: 124230:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2777.830567] Lustre: server umount lustre-MDT0001 complete [ 2782.742173] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 2784.176737] Key type lgssc unregistered [ 2784.344268] LNet: 124761:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2784.352319] LNetError: 124761:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2784.373137] LNet: Removed LNI 192.168.201.105@tcp [ 2784.812802] Key type .llcrypt unregistered [ 2784.814670] Key type ._llcrypt unregistered [ 2797.222640] Key type ._llcrypt registered [ 2797.223835] Key type .llcrypt registered [ 2797.300598] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 2797.748243] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2797.762063] alg: No test for adler32 (adler32-zlib) [ 2798.680033] Lustre: Lustre: Build Version: 2.17.0_RC1_3_g509e2bd [ 2798.798340] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 2800.415184] Key type lgssc registered [ 2801.075567] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2805.190236] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2805.202485] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2806.452901] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2808.347311] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2812.454614] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2812.662692] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2814.462293] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2817.230776] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2818.896967] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2822.007513] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2822.186542] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2825.197221] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2828.020138] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2831.405635] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:163 to 0x240000401:193) [ 2836.176377] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 2837.868204] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 2837.871800] Lustre: Skipped 2 previous similar messages [ 2840.049793] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 2840.053677] Lustre: Skipped 2 previous similar messages [ 2845.701063] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 2845.717176] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 2848.951281] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.201.5@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 2848.956368] Lustre: Skipped 2 previous similar messages [ 2855.436601] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.201.5@tcp 192.168.0.[2,10]@tcp [ 2858.443047] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.201.5/24@tcp in log lustre-MDT0000 [ 2858.447896] Lustre: Skipped 2 previous similar messages [ 2862.165731] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.5/24@tcp [ 2862.169483] Lustre: Skipped 1 previous similar message [ 2867.171306] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2867.173633] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2867.176531] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2867.196207] Lustre: Skipped 1 previous similar message [ 2871.348224] LustreError: 129975:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2871.414329] Lustre: server umount lustre-OST0000 complete [ 2874.337289] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2874.346871] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2874.355150] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2874.358317] Lustre: Skipped 1 previous similar message [ 2877.410755] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2877.411622] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2879.450083] LustreError: 130176:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2879.453884] LustreError: 130176:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2879.506727] Lustre: server umount lustre-MDT0000 complete [ 2881.426977] LustreError: 126115:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764736079 with bad export cookie 13792481343948686372 [ 2881.433934] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2881.435525] LustreError: 126115:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2881.516453] LustreError: 130377:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 2881.518975] LustreError: 130377:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2881.611996] Lustre: server umount lustre-MDT0001 complete [ 2887.018586] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 2888.596966] Key type lgssc unregistered [ 2888.775687] LNet: 130909:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2888.780466] LNetError: 130909:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2888.792756] LNet: Removed LNI 192.168.201.105@tcp [ 2889.272242] Key type .llcrypt unregistered [ 2889.273824] Key type ._llcrypt unregistered [ 2903.466505] Key type ._llcrypt registered [ 2903.468336] Key type .llcrypt registered [ 2903.529083] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 2904.037723] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2904.047421] alg: No test for adler32 (adler32-zlib) [ 2904.984835] Lustre: Lustre: Build Version: 2.17.0_RC1_3_g509e2bd [ 2905.105473] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 2906.719509] Key type lgssc registered [ 2907.413204] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2911.659766] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2911.677893] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2912.946839] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 2912.959176] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.201.5/24@tcp [ 2912.966567] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2914.841481] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2919.609711] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2919.863910] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 2919.879848] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.5/24@tcp [ 2919.916739] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2922.097200] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2925.100530] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2927.111427] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2930.680370] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2930.874971] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2933.583476] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2936.383292] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2938.036812] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:201 to 0x240000401:225) [ 2951.150269] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 2951.156550] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2951.163772] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2952.949984] LustreError: 134753:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2953.033946] Lustre: server umount lustre-OST0000 complete [ 2955.747949] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2955.753385] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2955.766866] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2955.793470] Lustre: Skipped 1 previous similar message [ 2960.865686] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2960.889344] Lustre: Skipped 1 previous similar message [ 2961.808679] LustreError: 134953:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2961.813963] LustreError: 134953:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2962.040061] Lustre: server umount lustre-MDT0000 complete [ 2965.990548] LustreError: 134954:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2966.002819] LustreError: 134954:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2966.472472] LustreError: 132411:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764736164 with bad export cookie 10615015093275067009 [ 2966.477788] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2966.500150] LustreError: 132411:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2975.720639] LustreError: 135156:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 2975.723552] LustreError: 135156:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2975.885121] Lustre: server umount lustre-MDT0001 complete [ 2986.593835] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 2989.409471] Key type lgssc unregistered [ 2989.720414] LNet: 135689:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2989.734508] LNetError: 135689:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2989.761132] LNet: Removed LNI 192.168.201.105@tcp [ 2990.625585] Key type .llcrypt unregistered [ 2990.628403] Key type ._llcrypt unregistered [ 3015.845941] Key type ._llcrypt registered [ 3015.848968] Key type .llcrypt registered [ 3015.941155] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [ 3026.958158] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 3027.667894] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3027.754959] alg: No test for adler32 (adler32-zlib) [ 3028.854555] Lustre: Lustre: Build Version: 2.17.0_RC1_3_g509e2bd [ 3029.050391] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 3030.724991] Key type lgssc registered [ 3031.554796] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3037.684658] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3042.689464] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3047.696494] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3052.575891] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3057.575244] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3057.630788] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3057.643907] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3058.851970] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3058.877776] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3058.922829] Lustre: lustre-MDT0000: new disk, initializing [ 3058.996541] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3059.022928] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3061.625524] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 3069.763970] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3069.815286] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3069.874375] Lustre: 139045:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3069.917493] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3069.920313] Lustre: Skipped 1 previous similar message [ 3069.975509] Lustre: lustre-MDT0001: new disk, initializing [ 3070.062842] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3070.111066] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3070.119084] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3072.586429] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 3077.963866] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3080.344840] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3084.910371] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3084.946978] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3085.107199] Lustre: lustre-OST0000: new disk, initializing [ 3085.111491] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3085.185196] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3089.244815] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 3090.993827] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3091.006755] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3091.102835] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3094.900990] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3098.036419] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3098.242724] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3101.452601] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3101.619292] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3106.792362] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3106.793896] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3106.796684] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3106.803313] Lustre: Skipped 1 previous similar message [ 3106.807022] LustreError: Skipped 1 previous similar message [ 3109.044196] LustreError: 141980:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3109.154824] Lustre: server umount lustre-OST0000 complete [ 3111.910780] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3111.917071] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3111.928669] Lustre: Skipped 1 previous similar message [ 3111.938788] Lustre: Skipped 2 previous similar messages [ 3117.024706] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3117.031919] Lustre: Skipped 1 previous similar message [ 3122.146216] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3122.148618] Lustre: Skipped 1 previous similar message [ 3126.239150] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3126.308660] LustreError: 142181:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3126.314205] LustreError: 142181:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3126.396413] Lustre: server umount lustre-MDT0000 complete [ 3128.692405] LustreError: 139036:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764736327 with bad export cookie 16326891164293342471 [ 3128.692405] LustreError: 139824:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764736327 with bad export cookie 16326891164293342471 [ 3128.692893] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3128.698951] LustreError: 139824:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3128.803186] LustreError: 142383:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3128.805513] LustreError: 142383:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3128.934069] Lustre: server umount lustre-MDT0001 complete [ 3134.494462] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3134.813591] LustreError: 142859:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3134.835438] LustreError: 142859:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3134.864664] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3137.521105] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 3140.065753] LustreError: 142859:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3142.956740] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3143.187694] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3145.431272] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 3148.708627] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3150.934878] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3155.286929] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3155.494559] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3159.833689] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 3163.766193] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3166.882796] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3167.025149] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3170.079847] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3170.256839] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3176.394220] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3176.447584] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3176.574640] Lustre: lustre-OST0001: new disk, initializing [ 3176.579608] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3176.665678] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3181.327950] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 3182.205598] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3182.209742] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3182.392898] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3188.182967] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3188.438513] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3191.846725] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3192.099992] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3194.757755] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff890718456800.ost_server_uuid 50 [ 3195.918616] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff890718456800.ost_server_uuid in FULL state after 0 sec [ 3203.041590] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3203.043886] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3203.046640] LustreError: Skipped 1 previous similar message [ 3203.058796] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3208.373811] LustreError: 147102:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 3208.376770] LustreError: 147102:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3208.426336] Lustre: server umount lustre-OST0001 complete [ 3213.282401] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3213.290774] LustreError: 144457:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3213.291685] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3213.291690] Lustre: Skipped 3 previous similar messages [ 3213.291771] Lustre: Skipped 2 previous similar messages [ 3213.313900] LustreError: 144457:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3217.075566] LustreError: 147303:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3217.079662] LustreError: 147303:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3217.153250] Lustre: server umount lustre-OST0000 complete [ 3219.935992] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3219.942365] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3225.576827] LustreError: 147504:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 3225.580485] LustreError: 147504:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3225.648290] Lustre: server umount lustre-MDT0000 complete [ 3228.311133] LustreError: 142838:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764736426 with bad export cookie 16326891164293343353 [ 3228.314657] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3228.325294] LustreError: 142838:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3228.550861] Lustre: server umount lustre-MDT0001 complete [ 3242.173494] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 3244.293801] Key type lgssc unregistered [ 3244.493853] LNet: 148699:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3244.501482] LNetError: 148699:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3244.516108] LNet: Removed LNI 192.168.201.105@tcp [ 3245.060562] Key type .llcrypt unregistered [ 3245.066239] Key type ._llcrypt unregistered [ 3269.179935] Key type ._llcrypt registered [ 3269.181910] Key type .llcrypt registered [ 3269.252098] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [ 3279.113075] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 3279.602807] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3279.631503] alg: No test for adler32 (adler32-zlib) [ 3280.557887] Lustre: Lustre: Build Version: 2.17.0_RC1_3_g509e2bd [ 3280.691560] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 3282.319193] Key type lgssc registered [ 3283.136776] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3289.419476] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3293.941401] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3298.451294] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3302.728191] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3307.015589] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3307.055878] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3307.066859] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3308.242099] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3308.272696] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3308.326683] Lustre: lustre-MDT0000: new disk, initializing [ 3308.422389] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3308.439357] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3311.082296] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 3317.878535] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3317.924641] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3317.995950] Lustre: 152624:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3318.057772] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3318.063259] Lustre: Skipped 1 previous similar message [ 3318.114400] Lustre: lustre-MDT0001: new disk, initializing [ 3318.190810] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3318.216790] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3318.222296] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3320.722307] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 3325.671450] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3327.834831] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3332.211231] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3332.288700] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3332.464862] Lustre: lustre-OST0000: new disk, initializing [ 3332.470314] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3332.543796] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3336.054833] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 3340.314847] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3340.322208] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3340.399795] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3341.388763] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3344.221518] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3344.353257] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3347.097261] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3347.266525] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3350.500423] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3350.502504] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3350.507959] LustreError: Skipped 1 previous similar message [ 3350.509331] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3350.515493] Lustre: Skipped 1 previous similar message [ 3350.523644] Lustre: Skipped 1 previous similar message [ 3354.563534] LustreError: 155567:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3354.652733] Lustre: server umount lustre-OST0000 complete [ 3359.203258] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3359.207056] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3359.216443] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3360.738837] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3360.740171] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3360.753809] Lustre: Skipped 1 previous similar message [ 3363.343045] LustreError: 155768:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3363.346132] LustreError: 155768:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3363.394595] Lustre: server umount lustre-MDT0000 complete [ 3365.681091] LustreError: 154373:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764736564 with bad export cookie 7329347001928417642 [ 3365.685446] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3365.688849] LustreError: 154373:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3365.797031] LustreError: 155969:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3365.802128] LustreError: 155969:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3365.909702] Lustre: server umount lustre-MDT0001 complete [ 3370.879058] Lustre: DEBUG MARKER: == conf-sanity test 47a: server restart does not lose lru_resize ========================================================== 23:36:08 (1764736568) [ 3381.484411] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [ 3388.482608] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 3394.475406] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3398.354134] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3402.072033] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3406.076823] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3410.490522] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3410.536539] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3410.708289] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3410.736596] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3410.790573] Lustre: lustre-MDT0000: new disk, initializing [ 3410.844077] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3410.857152] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3413.182899] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 3420.655414] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3420.716310] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3420.776518] Lustre: 158958:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3420.843071] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3420.853739] Lustre: Skipped 1 previous similar message [ 3420.913985] Lustre: lustre-MDT0001: new disk, initializing [ 3421.014132] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3421.051219] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3421.061428] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3423.717203] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 3429.099025] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3431.293908] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3435.289719] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3435.333741] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3435.484263] Lustre: lustre-OST0000: new disk, initializing [ 3435.488778] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3435.547311] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3438.727270] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 3443.234797] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3443.241163] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3443.296312] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3443.430129] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3446.164713] Lustre: Failing over lustre-OST0000 [ 3446.207086] LustreError: 161400:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3446.227824] LustreError: 161400:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3446.292842] Lustre: server umount lustre-OST0000 complete [ 3461.474478] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3461.673097] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3461.696576] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3461.713494] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 3463.784423] Lustre: 150837:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1764736646/real 1764736646] req@00000000747088cf x1850460318264960/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1764736662 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3465.130349] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 3465.312768] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 3465.631234] Lustre: 150840:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1764736648/real 1764736648] req@00000000925fb0fe x1850460318265728/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1764736664 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3465.647936] Lustre: 150840:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 3466.745782] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 3466.747156] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 3467.552594] Lustre: Failing over lustre-MDT0000 [ 3467.651163] LustreError: 162373:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3467.654206] LustreError: 162373:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3467.715256] Lustre: server umount lustre-MDT0000 complete [ 3468.577063] Lustre: 150840:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1764736651/real 1764736651] req@00000000b2eb2782 x1850460318266752/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1764736667 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3471.841104] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3471.849030] Lustre: Skipped 3 previous similar messages [ 3473.887346] Lustre: 150837:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1764736656/real 1764736656] req@0000000041eeb372 x1850460318267904/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1764736672 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3475.525606] LustreError: 158965:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3475.542090] LustreError: 158965:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 3476.962442] LustreError: 161516:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3476.968787] LustreError: 161516:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3480.640487] LustreError: 158964:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3483.554979] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3483.639066] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3483.647290] Lustre: 158961:0:(mgc_request.c:1910:mgc_process_log()) MGC192.168.201.105@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 3483.871029] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3484.784630] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3486.580567] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 3489.263934] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3489.270878] Lustre: Skipped 1 previous similar message [ 3489.276877] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 3489.309596] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 3492.323199] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3492.326987] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3492.333133] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3498.109372] LustreError: 163257:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3498.112295] LustreError: 163257:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3498.174916] Lustre: server umount lustre-OST0000 complete [ 3504.609849] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3504.610023] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3504.613244] Lustre: Skipped 3 previous similar messages [ 3514.848389] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3514.852609] Lustre: Skipped 3 previous similar messages [ 3515.359389] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3515.409397] LustreError: 163459:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 3515.411688] LustreError: 163459:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3515.467764] Lustre: server umount lustre-MDT0000 complete [ 3518.946298] LustreError: 158965:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3518.988480] LustreError: 158965:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 3519.020973] LustreError: 158950:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764736717 with bad export cookie 7329347001928419728 [ 3519.027662] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3519.041492] LustreError: 158950:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3519.370547] Lustre: server umount lustre-MDT0001 complete [ 3535.087985] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 3539.186356] Key type lgssc unregistered [ 3539.544293] LNet: 164195:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3539.552470] LNetError: 164195:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3539.576653] LNet: Removed LNI 192.168.201.105@tcp [ 3540.997092] Key type .llcrypt unregistered [ 3541.002967] Key type ._llcrypt unregistered [ 3580.937253] Key type ._llcrypt registered [ 3580.939352] Key type .llcrypt registered [ 3581.040278] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 3581.717245] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3581.747666] alg: No test for adler32 (adler32-zlib) [ 3582.762248] Lustre: Lustre: Build Version: 2.17.0_RC1_3_g509e2bd [ 3582.945095] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 3584.607193] Key type lgssc registered [ 3585.729600] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3592.154512] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3592.168541] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3593.581777] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3596.487777] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 3602.085634] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3602.924308] LustreError: 165709:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3602.939043] LustreError: 165709:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3605.299129] LustreError: 165738:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 3605.308515] LustreError: 165738:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 3605.314428] LustreError: 165738:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 11, retries 0, failed: rc = -5 [ 3605.458432] LustreError: 166261:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3605.564239] Lustre: server umount lustre-MDT0000 complete [ 3615.980403] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [ 3622.350178] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 3629.373620] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3634.332844] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3639.266656] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3644.055466] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3648.989399] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3649.040716] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3649.192428] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3649.226920] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3649.282379] Lustre: lustre-MDT0000: new disk, initializing [ 3649.354844] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3649.380858] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3652.522859] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 3660.481345] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3660.532703] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3660.628916] Lustre: 168905:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3660.686098] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3660.688820] Lustre: Skipped 1 previous similar message [ 3660.734599] Lustre: lustre-MDT0001: new disk, initializing [ 3660.820180] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3660.852947] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3660.858162] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3663.450622] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 3668.969693] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3672.294899] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3677.281956] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3677.366633] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3677.550815] Lustre: lustre-OST0000: new disk, initializing [ 3677.557768] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3677.629862] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3677.776807] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3677.784936] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3677.892616] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3681.996777] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 3687.747172] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3690.406264] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 3691.548370] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 23:41:29 (1764736889) [ 3872.735552] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3872.735827] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3872.743321] Lustre: Skipped 2 previous similar messages [ 3872.746480] Lustre: Skipped 1 previous similar message [ 3877.139523] LustreError: 172419:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3877.143058] LustreError: 172419:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3877.205678] Lustre: server umount lustre-MDT0000 complete [ 3877.857917] LustreError: 168910:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3877.870083] LustreError: 168910:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3879.319954] LustreError: 168897:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764737077 with bad export cookie 9953110799700553451 [ 3879.320851] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3879.325749] LustreError: 168897:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3882.975580] LustreError: 168910:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3882.975879] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3883.000951] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3883.004262] Lustre: Skipped 1 previous similar message [ 3885.024660] LustreError: 172298:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3885.024673] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3885.454322] LustreError: 172622:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3885.461170] LustreError: 172622:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3885.543117] Lustre: server umount lustre-MDT0001 complete [ 3893.936596] LustreError: 172826:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 3893.941417] LustreError: 172826:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3894.012306] Lustre: server umount lustre-OST0000 complete [ 3901.178739] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [ 3907.247467] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 3913.327635] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3917.368949] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3920.599086] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3924.083898] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3927.875848] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3927.910562] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3928.040137] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3928.068530] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3928.123574] Lustre: lustre-MDT0000: new disk, initializing [ 3928.195544] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3928.208181] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3930.458492] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 3936.503838] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3936.541200] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3936.597614] Lustre: 175163:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3936.638141] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3936.641260] Lustre: Skipped 1 previous similar message [ 3936.683232] Lustre: lustre-MDT0001: new disk, initializing [ 3936.726275] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3936.749651] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3936.756101] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3939.202613] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 3944.072978] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3946.143319] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3949.980453] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3950.018360] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3950.186489] Lustre: lustre-OST0000: new disk, initializing [ 3950.190189] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3953.302953] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 3956.246355] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3956.258428] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3956.344946] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3957.786239] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3960.190744] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3960.306532] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3962.252232] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3962.351111] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3966.435312] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3966.435648] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3966.442950] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3966.446559] LustreError: Skipped 1 previous similar message [ 3969.270138] LustreError: 178106:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3969.278976] LustreError: 178106:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3969.368766] Lustre: server umount lustre-OST0000 complete [ 3971.552184] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3971.560563] Lustre: Skipped 1 previous similar message [ 3971.563814] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3971.567653] Lustre: Skipped 2 previous similar messages [ 3977.381350] LustreError: 178307:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3977.384518] LustreError: 178307:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3977.437164] Lustre: server umount lustre-MDT0000 complete [ 3979.475995] LustreError: 175954:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764737177 with bad export cookie 9953110799700712288 [ 3979.476636] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3979.483227] LustreError: 175954:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3979.679307] Lustre: server umount lustre-MDT0001 complete [ 3984.486363] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 23:46:22 (1764737182) [ 3992.603712] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [ 3997.250908] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 4002.050643] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4005.531559] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4008.614476] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4012.000987] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4015.374646] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4015.416888] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4015.554755] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4015.584575] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4015.630159] Lustre: lustre-MDT0000: new disk, initializing [ 4015.699250] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4015.705585] Lustre: Skipped 1 previous similar message [ 4015.715261] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4017.704598] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 4023.105488] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4023.141235] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4023.193578] Lustre: 181495:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4023.347636] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4025.092807] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 4028.886825] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4030.601368] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4033.702423] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4033.731277] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4034.233348] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4034.306339] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4036.818829] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 4040.960370] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4044.770450] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4044.772251] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4044.774161] LustreError: Skipped 1 previous similar message [ 4044.778516] Lustre: Skipped 1 previous similar message [ 4044.781449] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4044.785778] Lustre: Skipped 3 previous similar messages [ 4050.674475] LustreError: 184043:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4050.677555] LustreError: 184043:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 4050.722193] Lustre: server umount lustre-OST0000 complete [ 4053.983907] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4053.986818] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4058.539152] Lustre: server umount lustre-MDT0000 complete [ 4060.129887] LustreError: 183194:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4060.137678] LustreError: 183194:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4060.143612] LustreError: 181486:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764737258 with bad export cookie 9953110799700713170 [ 4060.147512] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4060.148351] LustreError: 181486:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4060.294579] Lustre: server umount lustre-MDT0001 complete [ 4063.953405] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 23:47:42 (1764737262) [ 4071.579845] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [ 4075.812995] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 4080.197996] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4083.299075] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4086.183452] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4089.156794] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4092.385441] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4092.421766] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4092.544361] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4092.566253] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4092.569036] Lustre: Skipped 3 previous similar messages [ 4092.610685] Lustre: lustre-MDT0000: new disk, initializing [ 4092.613027] Lustre: Skipped 2 previous similar messages [ 4092.656630] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4092.659344] Lustre: Skipped 2 previous similar messages [ 4092.668630] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4092.673000] Lustre: Skipped 2 previous similar messages [ 4094.319153] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 4099.246293] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4099.282121] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4099.337101] Lustre: 187432:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4099.479470] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4101.248752] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 4104.783123] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4106.205225] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4108.840043] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4108.867343] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4110.840777] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4111.252720] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 4114.904851] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4121.057221] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4121.059638] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4121.061449] LustreError: Skipped 1 previous similar message [ 4121.062165] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4121.067382] Lustre: Skipped 2 previous similar messages [ 4121.069452] Lustre: Skipped 5 previous similar messages [ 4123.324934] LustreError: 189975:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4123.336742] LustreError: 189975:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 4123.399805] Lustre: server umount lustre-OST0000 complete [ 4131.524341] Lustre: server umount lustre-MDT0000 complete [ 4133.308980] LustreError: 188218:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764737331 with bad export cookie 9953110799700714255 [ 4133.312966] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4133.313986] LustreError: 188218:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4133.472841] Lustre: server umount lustre-MDT0001 complete [ 4137.914529] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 4139.201905] Key type lgssc unregistered [ 4139.359454] LNet: 190909:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4139.363392] LNetError: 190909:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4139.371605] LNet: Removed LNI 192.168.201.105@tcp [ 4139.743882] Key type .llcrypt unregistered [ 4139.745584] Key type ._llcrypt unregistered [ 4151.651066] Key type ._llcrypt registered [ 4151.652322] Key type .llcrypt registered [ 4151.722995] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 4152.194480] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4152.241109] alg: No test for adler32 (adler32-zlib) [ 4153.150278] Lustre: Lustre: Build Version: 2.17.0_RC1_3_g509e2bd [ 4153.270624] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 4154.863474] Key type lgssc registered [ 4155.385132] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4158.770755] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4158.779177] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4159.967630] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4161.626217] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 4165.109329] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4165.268335] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4166.949391] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 4169.382661] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4171.015568] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4173.999482] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4174.162683] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4176.615599] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 4179.052207] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4185.448206] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4206.052220] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4206.052631] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4206.056017] LustreError: Skipped 1 previous similar message [ 4206.057072] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4206.061308] Lustre: Skipped 1 previous similar message [ 4207.983082] LustreError: 194553:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4208.028341] Lustre: server umount lustre-OST0000 complete [ 4211.169116] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4211.176810] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4211.179882] Lustre: Skipped 2 previous similar messages [ 4215.753048] LustreError: 194756:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4215.756030] LustreError: 194756:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4215.788452] Lustre: server umount lustre-MDT0000 complete [ 4216.288409] LustreError: 192281:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4216.295659] LustreError: 192281:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4217.432629] LustreError: 192265:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764737415 with bad export cookie 10191713691066129365 [ 4217.435846] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4217.437781] LustreError: 192265:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4217.511663] LustreError: 194960:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4217.513853] LustreError: 194960:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4217.586878] Lustre: server umount lustre-MDT0001 complete [ 4222.032873] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 4223.376529] Key type lgssc unregistered [ 4223.529844] LNet: 195491:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4223.534191] LNetError: 195491:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4223.541402] LNet: Removed LNI 192.168.201.105@tcp [ 4223.907864] Key type .llcrypt unregistered [ 4223.909274] Key type ._llcrypt unregistered [ 4235.491253] Key type ._llcrypt registered [ 4235.492830] Key type .llcrypt registered [ 4235.544182] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 4236.186339] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4236.205377] alg: No test for adler32 (adler32-zlib) [ 4237.134737] Lustre: Lustre: Build Version: 2.17.0_RC1_3_g509e2bd [ 4237.261910] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 4238.863428] Key type lgssc registered [ 4239.456233] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4243.250131] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4243.263123] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4244.569424] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4246.349127] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 4250.259474] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4250.423381] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4252.274292] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 4254.733532] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4256.294977] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4259.086082] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4259.268074] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4261.676429] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 4263.855937] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4269.545673] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 4279.375053] Lustre: lustre-OST0000: Not available for connect from 192.168.201.5@tcp (stopping) [ 4279.777498] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4279.780706] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4279.785683] Lustre: Skipped 1 previous similar message [ 4283.261775] LustreError: 199134:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4283.327389] Lustre: server umount lustre-OST0000 complete [ 4286.072390] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff890719c6f800.ost_server_uuid 50 [ 4287.031572] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 4299.618162] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4299.619800] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4299.629492] Lustre: Skipped 1 previous similar message [ 4299.634255] Lustre: Skipped 3 previous similar messages [ 4303.701906] LustreError: 199481:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4303.705611] LustreError: 199481:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4303.764426] Lustre: server umount lustre-MDT0000 complete [ 4304.866375] LustreError: 196867:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4304.876238] LustreError: 196867:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4305.874313] LustreError: 196846:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764737504 with bad export cookie 11816814738623449725 [ 4305.877687] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4305.879478] LustreError: 196846:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4305.987720] LustreError: 199685:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4305.992385] LustreError: 199685:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4306.083707] Lustre: server umount lustre-MDT0001 complete [ 4311.063736] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 23:51:48 (1764737508) [ 4314.659453] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4314.912099] LustreError: 200355:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4314.950382] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4317.028732] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 4320.225888] LustreError: 200355:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4321.617952] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4321.827719] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4323.642263] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 4326.216916] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4327.928319] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4330.912511] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4331.070723] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4333.775877] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 4336.579726] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4340.154595] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4340.195209] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4340.332129] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 4340.342858] Lustre: lustre-OST0001: new disk, initializing [ 4340.347405] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4343.264815] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 4347.421129] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4347.424911] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4347.485792] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4347.590797] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4350.943852] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4350.952519] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4350.959429] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4353.101801] Lustre: lustre-OST0000: Not available for connect from 192.168.201.5@tcp (stopping) [ 4353.105438] Lustre: Skipped 2 previous similar messages [ 4355.182121] LustreError: 203559:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4355.184471] LustreError: 203559:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4355.213848] Lustre: server umount lustre-OST0000 complete [ 4357.605566] LustreError: 201954:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4357.611847] LustreError: 201954:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4358.182476] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4358.300801] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 4360.445607] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4360.575309] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 4362.720685] LustreError: 201951:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4362.728447] LustreError: 201951:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4372.962192] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4372.962896] LustreError: 201951:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4372.963341] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4372.963349] Lustre: Skipped 1 previous similar message [ 4372.964810] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4372.966052] LustreError: Skipped 1 previous similar message [ 4372.972994] LustreError: 201951:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 4377.599996] LustreError: 204260:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 4377.603532] LustreError: 204260:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4377.698325] Lustre: server umount lustre-OST0001 complete [ 4383.200968] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4383.200988] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4383.204631] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4383.204641] Lustre: Skipped 1 previous similar message [ 4383.215085] Lustre: Skipped 2 previous similar messages [ 4385.943867] LustreError: 204461:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 4385.946176] LustreError: 204461:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4385.985694] Lustre: server umount lustre-MDT0000 complete [ 4387.882253] LustreError: 200335:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764737586 with bad export cookie 11816814738623450782 [ 4387.885389] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4387.888976] LustreError: 200335:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4388.067456] Lustre: server umount lustre-MDT0001 complete [ 4390.916896] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4392.723283] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4394.326966] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4396.151657] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4400.031845] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 23:53:17 (1764737597) [ 4404.250833] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4404.498253] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4404.536752] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4404.676967] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4404.680262] Lustre: Skipped 1 previous similar message [ 4407.209786] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 4411.999558] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4412.062566] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4412.082639] Lustre: 206153:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4414.348922] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 4416.993878] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4418.676226] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4421.857327] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4421.953392] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4424.729307] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 4427.171683] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4430.070075] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4430.107772] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 4432.007143] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 4432.021661] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 4432.785665] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 4435.301405] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4436.960730] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4436.964520] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4436.971032] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4436.973190] Lustre: Skipped 1 previous similar message [ 4438.519111] LustreError: 209342:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4438.522079] LustreError: 209342:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 4438.546343] Lustre: server umount lustre-OST0000 complete [ 4440.653218] LustreError: 207782:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4452.837370] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4452.840457] LustreError: Skipped 2 previous similar messages [ 4455.972159] Lustre: server umount lustre-OST0001 complete [ 4457.952824] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4457.957575] Lustre: Skipped 4 previous similar messages [ 4463.867297] Lustre: server umount lustre-MDT0000 complete [ 4465.807227] LustreError: 206145:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764737664 with bad export cookie 11816814738623452182 [ 4465.811631] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4465.812680] LustreError: 206145:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4465.993665] Lustre: server umount lustre-MDT0001 complete [ 4468.781628] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4470.806623] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4472.624241] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4474.415103] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4477.712906] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 23:54:35 (1764737675) [ 4484.415577] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [ 4488.360630] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 4492.971105] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4495.873776] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4498.370141] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4501.124977] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4503.819058] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4503.851110] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4503.968130] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4503.981952] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4504.024622] Lustre: lustre-MDT0000: new disk, initializing [ 4504.059361] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4504.062025] Lustre: Skipped 3 previous similar messages [ 4504.071602] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4505.712855] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 4510.705025] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4510.732990] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4510.775596] Lustre: 213770:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4510.810761] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4510.813622] Lustre: Skipped 1 previous similar message [ 4510.860845] Lustre: lustre-MDT0001: new disk, initializing [ 4510.929921] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4510.935819] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4512.714458] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 4516.597750] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4518.374347] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4521.739698] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4521.776681] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4521.942323] Lustre: lustre-OST0000: new disk, initializing [ 4521.944864] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4524.703326] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 4528.932294] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4530.194768] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4530.201931] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4530.258468] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4531.051128] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4531.169738] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4533.234902] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4533.348895] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4535.265832] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4535.272874] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4535.276243] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4535.281454] Lustre: Skipped 1 previous similar message [ 4535.284188] Lustre: Skipped 8 previous similar messages [ 4536.369186] LustreError: 216708:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4536.372699] LustreError: 216708:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 4536.441400] Lustre: server umount lustre-OST0000 complete [ 4544.434898] Lustre: server umount lustre-MDT0000 complete [ 4545.504393] LustreError: 213781:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4545.511881] LustreError: 213781:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 4546.297778] LustreError: 213760:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764737744 with bad export cookie 11816814738623453855 [ 4546.301068] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4546.302773] LustreError: 213760:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4546.479434] Lustre: server umount lustre-MDT0001 complete [ 4550.431560] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4552.737102] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 4556.703677] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4558.679266] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 4561.152161] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4562.673834] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4565.519463] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4568.235868] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 4570.963643] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4572.986373] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4576.199491] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 4578.244550] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4578.356678] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4581.347689] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4581.351953] LustreError: Skipped 1 previous similar message [ 4585.431620] Lustre: server umount lustre-OST0000 complete [ 4588.194674] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4590.032738] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4614.470843] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4614.636778] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4614.640963] Lustre: Skipped 5 previous similar messages [ 4617.072668] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 4619.418046] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4620.122385] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4620.773774] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4620.780384] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4620.780448] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4625.891466] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4625.896049] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4625.898254] Lustre: Skipped 5 previous similar messages [ 4625.901071] Lustre: Skipped 5 previous similar messages [ 4627.443157] LustreError: 221988:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4627.445864] LustreError: 221988:0:(obd_class.h:479:obd_check_dev()) Skipped 21 previous similar messages [ 4627.493506] Lustre: server umount lustre-OST0000 complete [ 4635.244384] Lustre: server umount lustre-MDT0000 complete [ 4636.128489] LustreError: 217585:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4636.135363] LustreError: 217585:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4636.766265] LustreError: 217565:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764737835 with bad export cookie 11816814738623454723 [ 4636.770034] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4636.771131] LustreError: 217565:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4636.904192] Lustre: server umount lustre-MDT0001 complete [ 4640.280531] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 23:57:18 (1764737838) [ 4643.065571] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4644.996197] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 4648.683472] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4650.549694] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 4652.807923] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4654.302944] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4657.233692] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4659.746354] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 4662.055666] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4663.719626] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4667.922149] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 4 sec [ 4669.594426] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4669.700533] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4672.337227] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4672.365654] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4672.427635] Lustre: lustre-OST0001: new disk, initializing [ 4672.430068] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4674.693024] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 4678.284519] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4679.979677] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4681.733620] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4681.739046] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4681.771423] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4682.121193] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 4683.849700] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4683.975675] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4686.819673] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4686.823267] LustreError: Skipped 3 previous similar messages [ 4690.877806] Lustre: server umount lustre-OST0001 complete [ 4693.235220] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4693.331297] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 4694.909738] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4694.998741] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 4719.295754] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4721.782099] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 4724.091720] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4724.710975] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 4724.716801] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 4724.717043] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4724.726369] Lustre: Skipped 2 previous similar messages [ 4724.763619] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4727.617026] Lustre: server umount lustre-OST0001 complete [ 4746.265751] Lustre: server umount lustre-OST0000 complete [ 4755.428140] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4755.431030] Lustre: Skipped 12 previous similar messages [ 4762.079265] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4762.129993] LustreError: 229385:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 4762.133214] LustreError: 229385:0:(obd_class.h:479:obd_check_dev()) Skipped 29 previous similar messages [ 4762.172772] Lustre: server umount lustre-MDT0000 complete [ 4763.871738] LustreError: 229084:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764737962 with bad export cookie 11816814738623455675 [ 4763.876044] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4763.876423] LustreError: 229084:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4770.050077] Lustre: server umount lustre-MDT0001 complete [ 4772.389441] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4773.991848] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4775.429523] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4777.054090] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4779.801422] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 23:59:37 (1764737977) [ 4782.668744] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4782.807253] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4782.828515] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4782.924461] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4782.927923] Lustre: Skipped 5 previous similar messages [ 4784.591281] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 4788.402710] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4788.467502] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4788.488565] Lustre: 231079:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4790.336599] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 4792.584703] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4794.084641] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4796.794333] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4796.882347] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4799.054724] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 4801.099566] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4810.579779] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4810.608953] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 4813.002527] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 4815.068867] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4817.755878] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 2 sec [ 4819.216911] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4819.295307] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4820.769676] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4820.855185] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4822.249200] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff890710735000.ost_server_uuid 50 [ 4822.818907] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff890710735000.ost_server_uuid in FULL state after 0 sec [ 4823.175204] Lustre: Permanently deactivating lustre-OST0001 [ 4823.180651] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 4824.063947] Lustre: Permanently reactivating lustre-OST0001 [ 4829.154252] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4829.157682] LustreError: Skipped 4 previous similar messages [ 4829.158941] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4829.163651] Lustre: Skipped 10 previous similar messages [ 4831.229534] Lustre: server umount lustre-OST0001 complete [ 4834.276829] LustreError: 234979:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4834.282276] LustreError: 234979:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 28 previous similar messages [ 4838.740914] Lustre: server umount lustre-OST0000 complete [ 4846.305759] Lustre: server umount lustre-MDT0000 complete [ 4847.770762] LustreError: 231070:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764738046 with bad export cookie 11816814738623456949 [ 4847.773308] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4847.775739] LustreError: 231070:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4847.903410] Lustre: server umount lustre-MDT0001 complete [ 4850.128667] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4851.704723] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4853.169191] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4854.758863] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4857.521518] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 00:00:55 (1764738055) [ 4857.999637] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4861.127391] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4861.297479] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4861.315116] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4861.319235] Lustre: Skipped 5 previous similar messages [ 4863.310527] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 4866.940552] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4867.001450] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4867.017508] Lustre: 237221:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4868.821178] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 4871.276404] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4872.885641] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4876.051886] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4876.094144] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4876.211472] Lustre: Permanently deactivating lustre-OST0000 [ 4876.218653] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 4878.787719] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 4880.381639] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4881.234461] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4884.725535] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4887.696703] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 4888.050972] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 4890.133361] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4891.006326] Lustre: Permanently reactivating lustre-OST0000 [ 4897.788968] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 4897.791588] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4897.794875] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4897.815821] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 4897.823028] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 4909.191761] Lustre: server umount lustre-OST0001 complete [ 4917.095566] Lustre: server umount lustre-OST0000 complete [ 4925.165917] Lustre: server umount lustre-MDT0000 complete [ 4926.674721] LustreError: 240383:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764738125 with bad export cookie 11816814738623458531 [ 4926.679480] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4926.681358] LustreError: 240383:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4926.816443] Lustre: server umount lustre-MDT0001 complete [ 4930.960471] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 4932.238788] Key type lgssc unregistered [ 4932.389543] LNet: 241648:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4932.392745] LNetError: 241648:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4932.401422] LNet: Removed LNI 192.168.201.105@tcp [ 4932.778850] Key type .llcrypt unregistered [ 4932.780313] Key type ._llcrypt unregistered [ 4943.182217] Key type ._llcrypt registered [ 4943.184090] Key type .llcrypt registered [ 4943.242168] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 4943.842634] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4943.856647] alg: No test for adler32 (adler32-zlib) [ 4944.765727] Lustre: Lustre: Build Version: 2.17.0_RC1_3_g509e2bd [ 4944.895824] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 4946.503283] Key type lgssc registered [ 4947.065630] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4948.892701] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4952.423081] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4952.437235] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4953.723542] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4955.681800] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 4959.786685] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4959.824428] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4959.896618] Lustre: Found index 1 for lustre-MDT0001, updating log [ 4959.916672] Lustre: 243064:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4959.923674] Lustre: Permanently deactivating lustre-MDT0001 [ 4959.931567] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 4960.091605] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4962.004325] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 4962.290416] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 4962.295621] LustreError: 243106:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4962.303989] LustreError: 243106:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4962.309560] LustreError: 243106:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 8, retries 1, failed: rc = -108 [ 4964.577107] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4966.365591] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4970.173870] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4970.338635] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 4970.342287] LustreError: 244694:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4970.348096] LustreError: 244694:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4970.406560] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4973.505883] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 4975.594833] LustreError: 244693:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4975.603159] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 4976.162549] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4978.658081] LustreError: 244693:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4978.669792] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 4979.627083] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4979.785629] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4982.824867] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 4984.817459] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 4984.824367] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 4985.474729] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4987.330621] Lustre: Permanently reactivating lustre-MDT0001 [ 4987.338941] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 4987.343054] Lustre: Skipped 2 previous similar messages [ 4991.193837] Lustre: lustre-MDT0001: Client bf883f4a-8226-407e-bfae-292e6c57a91a (at 192.168.201.5@tcp) reconnecting [ 4993.444074] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4994.282683] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4996.715906] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 4996.828481] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 4998.643742] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 4998.748797] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 4999.883126] Lustre: Permanently deactivating lustre-MDT0001 [ 4999.892057] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 4999.895220] Lustre: Skipped 2 previous similar messages [ 5001.051540] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 5010.402480] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5010.404239] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5010.408226] Lustre: Skipped 3 previous similar messages [ 5010.410813] Lustre: Skipped 2 previous similar messages [ 5011.076390] LustreError: 247138:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 5011.117538] Lustre: server umount lustre-MDT0000 complete [ 5012.751549] LustreError: 243056:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764738211 with bad export cookie 7449455798985510771 [ 5012.755722] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5012.756346] LustreError: 243056:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5012.829548] LustreError: 247339:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 5012.832718] LustreError: 247339:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5012.904463] Lustre: server umount lustre-MDT0001 complete [ 5020.727151] LustreError: 247540:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 5020.730095] LustreError: 247540:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5020.752291] Lustre: server umount lustre-OST0000 complete [ 5026.453605] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 00:03:44 (1764738224) [ 5031.534911] LustreError: 248545:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 5031.539041] LustreError: 248545:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 5031.609837] Lustre: server umount lustre-OST0001 complete [ 5034.505625] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [ 5038.477817] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 5042.732628] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5045.509484] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5048.204852] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5050.945758] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5053.967488] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5053.992788] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5054.118052] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5054.123032] Lustre: Skipped 2 previous similar messages [ 5054.138328] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5054.186834] Lustre: lustre-MDT0000: new disk, initializing [ 5054.231975] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5054.242747] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5056.009046] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 5061.168373] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5061.198682] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5061.246043] Lustre: 250726:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5061.278871] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5061.282516] Lustre: Skipped 1 previous similar message [ 5061.326272] Lustre: lustre-MDT0001: new disk, initializing [ 5061.386809] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5061.391287] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5063.150447] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 5066.951578] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5068.521907] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5071.194064] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5071.225748] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5071.364343] Lustre: lustre-OST0000: new disk, initializing [ 5071.367519] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5071.409516] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5071.413884] Lustre: Skipped 1 previous similar message [ 5073.802796] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 5077.320777] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5081.098501] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5081.106218] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5081.152885] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5083.414639] LustreError: 250735:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 5088.436342] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5088.467600] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5088.544647] Lustre: lustre-OST0001: new disk, initializing [ 5088.547177] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5088.584811] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5089.100785] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5089.106598] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5089.159692] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5091.024514] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 5093.511055] LustreError: 250735:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 5094.500699] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5095.615250] LustreError: 251780:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 5096.173965] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 1 sec [ 5105.703153] LustreError: 251780:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 5109.731599] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5109.733229] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5109.738133] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5109.740418] LustreError: Skipped 1 previous similar message [ 5109.742680] Lustre: Skipped 2 previous similar messages [ 5112.557835] LustreError: 254224:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 5112.560900] LustreError: 254224:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5112.584401] Lustre: server umount lustre-OST0001 complete [ 5114.852615] LustreError: 252458:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5114.858670] LustreError: 252458:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5119.972121] LustreError: 252458:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5119.978421] LustreError: 252458:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5125.092528] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5125.095423] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5125.099551] Lustre: Skipped 2 previous similar messages [ 5125.102234] Lustre: Skipped 1 previous similar message [ 5130.211819] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5130.212058] LustreError: 254342:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5130.214594] Lustre: Skipped 1 previous similar message [ 5130.220523] LustreError: 254342:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5131.116837] LustreError: 254448:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5131.119851] LustreError: 254448:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5131.169841] Lustre: server umount lustre-OST0000 complete [ 5133.281710] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5133.286901] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5138.855664] Lustre: server umount lustre-MDT0000 complete [ 5140.275268] LustreError: 250718:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764738338 with bad export cookie 7449455798985513578 [ 5140.279262] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5140.279777] LustreError: 250718:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5146.489908] Lustre: server umount lustre-MDT0001 complete [ 5150.645650] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 5151.873268] Key type lgssc unregistered [ 5152.007510] LNet: 255383:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5152.010671] LNetError: 255383:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5152.017443] LNet: Removed LNI 192.168.201.105@tcp [ 5152.368979] Key type .llcrypt unregistered [ 5152.370855] Key type ._llcrypt unregistered [ 5165.003424] Key type ._llcrypt registered [ 5165.004817] Key type .llcrypt registered [ 5165.048148] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [ 5171.643246] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 5172.148770] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5172.157860] alg: No test for adler32 (adler32-zlib) [ 5173.021050] Lustre: Lustre: Build Version: 2.17.0_RC1_3_g509e2bd [ 5173.114105] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 5174.703157] Key type lgssc registered [ 5175.078670] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5178.464407] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5181.195718] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5183.635468] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5186.212744] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5189.096274] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5189.118362] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5189.125420] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5190.238995] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5190.261732] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5190.303288] Lustre: lustre-MDT0000: new disk, initializing [ 5190.334501] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5190.342418] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5191.880095] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 5196.610600] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5196.639527] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5196.680772] Lustre: 259306:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5196.715893] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5196.718853] Lustre: Skipped 1 previous similar message [ 5196.761865] Lustre: lustre-MDT0001: new disk, initializing [ 5196.819864] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5196.840918] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5196.845838] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5198.389871] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 5201.876770] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5203.234854] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5206.023239] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5206.053806] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5206.175320] Lustre: lustre-OST0000: new disk, initializing [ 5206.178448] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5206.222574] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5208.789286] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 5209.088079] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5209.094018] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5209.140500] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5212.531795] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5214.355690] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5214.450678] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5216.125406] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5216.222477] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5219.297532] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5219.298172] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5219.304145] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5219.309806] Lustre: Skipped 1 previous similar message [ 5223.154124] LustreError: 262244:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5223.219656] Lustre: server umount lustre-OST0000 complete [ 5227.489966] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5227.493718] LustreError: Skipped 1 previous similar message [ 5227.497624] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5227.503606] Lustre: Skipped 1 previous similar message [ 5227.506909] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5229.537178] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5229.537467] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5234.656809] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5234.660247] Lustre: Skipped 2 previous similar messages [ 5238.753915] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5238.758442] Lustre: Skipped 1 previous similar message [ 5239.263189] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5239.307796] LustreError: 262445:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5239.310920] LustreError: 262445:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5239.348476] Lustre: server umount lustre-MDT0000 complete [ 5241.036658] LustreError: 262447:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764738439 with bad export cookie 11300764440411407363 [ 5241.040663] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5241.042069] LustreError: 262447:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5241.121628] LustreError: 262649:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5241.124590] LustreError: 262649:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5241.195706] Lustre: server umount lustre-MDT0001 complete [ 5244.953457] Lustre: DEBUG MARKER: == conf-sanity test 52: check recovering objects from lost+found ========================================================== 00:07:23 (1764738443) [ 5247.876489] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5248.128375] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5249.834111] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 5253.504288] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5253.599912] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 5253.673435] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5255.415615] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 5257.754980] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5259.308280] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5262.121935] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5262.280975] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5264.666497] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 5266.817437] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5319.649983] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5319.652851] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5319.653347] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5319.655773] Lustre: Skipped 1 previous similar message [ 5322.415070] LustreError: 265662:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5322.417397] LustreError: 265662:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5322.465345] Lustre: server umount lustre-OST0000 complete [ 5324.084945] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 5328.706223] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5328.838124] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5330.830580] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 5332.651300] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5340.136910] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5340.141221] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5340.141233] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5340.147894] Lustre: Skipped 1 previous similar message [ 5370.849742] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5370.850057] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5370.851268] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5370.851273] Lustre: Skipped 1 previous similar message [ 5370.852829] LustreError: Skipped 1 previous similar message [ 5370.864146] Lustre: Skipped 1 previous similar message [ 5372.268128] LustreError: 267326:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5372.271554] LustreError: 267326:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5372.323876] Lustre: server umount lustre-OST0000 complete [ 5379.800660] Lustre: server umount lustre-MDT0000 complete [ 5381.088510] LustreError: 263319:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5381.094486] LustreError: 263319:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5381.106203] LustreError: 267529:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764738579 with bad export cookie 11300764440411408231 [ 5381.109652] LustreError: 267529:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5381.109863] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5381.171934] LustreError: 267730:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5381.173852] LustreError: 267730:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 5381.228853] Lustre: server umount lustre-MDT0001 complete [ 5384.868591] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 5385.971913] Key type lgssc unregistered [ 5386.107502] LNet: 268260:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5386.110781] LNetError: 268260:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5386.118409] LNet: Removed LNI 192.168.201.105@tcp [ 5386.450586] Key type .llcrypt unregistered [ 5386.452153] Key type ._llcrypt unregistered [ 5395.606640] Key type ._llcrypt registered [ 5395.608117] Key type .llcrypt registered [ 5395.650536] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 5396.042311] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5396.051413] alg: No test for adler32 (adler32-zlib) [ 5396.934843] Lustre: Lustre: Build Version: 2.17.0_RC1_3_g509e2bd [ 5397.034845] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 5398.623258] Key type lgssc registered [ 5399.050887] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5402.120254] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5402.129089] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5403.310122] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5404.639079] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 5407.700988] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5407.832273] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5409.251958] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 5411.110515] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5412.410839] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5414.972849] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5415.102725] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5417.141039] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 5419.071260] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5429.478333] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 5438.943956] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5438.946991] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5438.953058] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5439.968927] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5439.969095] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5439.971460] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5443.822149] LustreError: 272673:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5443.866184] Lustre: server umount lustre-OST0000 complete [ 5448.672072] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5448.674771] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5448.679824] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5448.682509] Lustre: Skipped 1 previous similar message [ 5451.327708] LustreError: 272875:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5451.330273] LustreError: 272875:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5451.361883] Lustre: server umount lustre-MDT0000 complete [ 5452.636583] LustreError: 269616:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764738651 with bad export cookie 5937935928341422397 [ 5452.640140] LustreError: 269616:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5452.640246] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5452.715727] LustreError: 273077:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5452.719024] LustreError: 273077:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5452.766419] Lustre: server umount lustre-MDT0001 complete [ 5456.321804] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 5457.405749] Key type lgssc unregistered [ 5457.533459] LNet: 273607:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5457.536294] LNetError: 273607:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5457.544340] LNet: Removed LNI 192.168.201.105@tcp [ 5457.846218] Key type .llcrypt unregistered [ 5457.847340] Key type ._llcrypt unregistered [ 5465.308693] Key type ._llcrypt registered [ 5465.310128] Key type .llcrypt registered [ 5465.355257] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 5465.815184] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5465.825102] alg: No test for adler32 (adler32-zlib) [ 5466.688131] Lustre: Lustre: Build Version: 2.17.0_RC1_3_g509e2bd [ 5466.780605] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 5468.367230] Key type lgssc registered [ 5468.782880] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5471.756873] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5471.764929] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5472.946769] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5474.506530] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 5477.721345] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5477.880253] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5479.379256] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 5481.325835] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5482.537842] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5484.838604] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5484.963826] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5486.899732] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 5488.726644] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5498.278327] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5503.462422] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5503.462613] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5503.464879] LustreError: Skipped 1 previous similar message [ 5503.470082] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5503.472970] Lustre: Skipped 1 previous similar message [ 5505.070109] LustreError: 277202:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5505.110839] Lustre: server umount lustre-OST0000 complete [ 5508.475660] LustreError: 277403:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5508.477701] LustreError: 277403:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5508.503527] Lustre: server umount lustre-MDT0000 complete [ 5508.577419] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5508.578734] LustreError: 274783:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5508.579859] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5508.582534] Lustre: Skipped 2 previous similar messages [ 5508.591134] LustreError: 274783:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5509.704049] LustreError: 276407:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764738708 with bad export cookie 18412262296281675680 [ 5509.706978] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5509.707882] LustreError: 276407:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5509.771599] LustreError: 277604:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5509.773416] LustreError: 277604:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5509.821257] Lustre: server umount lustre-MDT0001 complete [ 5513.305454] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 5514.318553] Key type lgssc unregistered [ 5514.456453] LNet: 278134:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5514.459646] LNetError: 278134:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5514.467337] LNet: Removed LNI 192.168.201.105@tcp [ 5514.772732] Key type .llcrypt unregistered [ 5514.774424] Key type ._llcrypt unregistered [ 5523.742288] Key type ._llcrypt registered [ 5523.743387] Key type .llcrypt registered [ 5523.785105] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 5524.118256] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5524.164550] alg: No test for adler32 (adler32-zlib) [ 5525.040073] Lustre: Lustre: Build Version: 2.17.0_RC1_3_g509e2bd [ 5525.132830] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 5526.719161] Key type lgssc registered [ 5527.086471] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5529.737215] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5529.743803] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5530.904140] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5532.196493] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 5535.026872] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5535.144556] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5536.484218] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 5538.323549] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5539.551742] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5541.884931] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5541.996281] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5543.881804] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 5545.650631] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5556.326686] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 5568.993373] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 5568.995820] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5569.001747] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5571.040347] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5571.043269] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5571.047258] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5572.066335] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5572.069197] Lustre: Skipped 1 previous similar message [ 5575.019973] LustreError: 282547:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5575.055740] Lustre: server umount lustre-OST0000 complete [ 5577.184620] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5577.190171] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5577.192494] Lustre: Skipped 1 previous similar message [ 5582.304504] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5582.306532] Lustre: Skipped 1 previous similar message [ 5582.397600] LustreError: 282750:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5582.400141] LustreError: 282750:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5582.427626] Lustre: server umount lustre-MDT0000 complete [ 5583.612403] LustreError: 279490:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764738782 with bad export cookie 13731899329855830557 [ 5583.614641] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5583.617292] LustreError: 279490:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5583.678958] LustreError: 282953:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5583.680766] LustreError: 282953:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5583.738702] Lustre: server umount lustre-MDT0001 complete [ 5587.190106] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 5588.255548] Key type lgssc unregistered [ 5588.383526] LNet: 283485:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5588.386685] LNetError: 283485:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5588.393393] LNet: Removed LNI 192.168.201.105@tcp [ 5588.708045] Key type .llcrypt unregistered [ 5588.709472] Key type ._llcrypt unregistered [ 5595.589485] Key type ._llcrypt registered [ 5595.590575] Key type .llcrypt registered [ 5595.629584] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 5595.976179] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5595.985383] alg: No test for adler32 (adler32-zlib) [ 5596.840815] Lustre: Lustre: Build Version: 2.17.0_RC1_3_g509e2bd [ 5596.926066] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 5598.511128] Key type lgssc registered [ 5598.830089] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5601.375050] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5601.381513] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5602.557258] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5603.907418] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 5606.787029] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5606.904753] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5608.190723] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 5610.018937] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5611.185112] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5613.427610] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5613.544380] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5615.311871] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 5617.017483] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5626.854428] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 5631.968659] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5631.970686] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5631.972512] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5631.975399] Lustre: Skipped 1 previous similar message [ 5636.910105] LustreError: 287110:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5636.947884] Lustre: server umount lustre-OST0000 complete [ 5642.208708] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5642.209200] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5642.212665] Lustre: Skipped 1 previous similar message [ 5642.215463] Lustre: Skipped 2 previous similar messages [ 5644.353586] LustreError: 287313:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5644.356217] LustreError: 287313:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5644.383739] Lustre: server umount lustre-MDT0000 complete [ 5645.623222] LustreError: 287315:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764738844 with bad export cookie 13421661023295719830 [ 5645.625655] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5645.628511] LustreError: 287315:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5645.701233] LustreError: 287516:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5645.704420] LustreError: 287516:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5645.771916] Lustre: server umount lustre-MDT0001 complete [ 5649.501293] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 5650.564697] Key type lgssc unregistered [ 5650.686369] LNet: 288046:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5650.689418] LNetError: 288046:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5650.696337] LNet: Removed LNI 192.168.201.105@tcp [ 5650.971232] Key type .llcrypt unregistered [ 5650.972639] Key type ._llcrypt unregistered [ 5654.665536] Key type ._llcrypt registered [ 5654.666617] Key type .llcrypt registered [ 5654.707860] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 5660.051609] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [ 5666.175789] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 5666.613455] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5666.638144] alg: No test for adler32 (adler32-zlib) [ 5667.496848] Lustre: Lustre: Build Version: 2.17.0_RC1_3_g509e2bd [ 5667.577867] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 5669.159228] Key type lgssc registered [ 5669.518119] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5672.544114] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5674.679697] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5676.822090] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5679.023225] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5681.445395] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5681.462600] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5681.468169] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5682.555259] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5682.569328] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5682.608658] Lustre: lustre-MDT0000: new disk, initializing [ 5682.636389] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5682.644180] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5683.957472] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 5688.232025] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5688.255576] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5688.291471] Lustre: 291529:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5688.312548] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5688.315127] Lustre: Skipped 1 previous similar message [ 5688.352801] Lustre: lustre-MDT0001: new disk, initializing [ 5688.392143] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5688.408105] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5688.412858] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5689.753470] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 5692.770940] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5694.002911] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5696.344269] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5696.369403] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5696.466471] Lustre: lustre-OST0000: new disk, initializing [ 5696.468316] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5696.499671] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5697.537442] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5697.541272] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5697.575385] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5698.409360] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 5701.475620] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5702.823359] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5702.900617] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5704.235101] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5704.310776] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5707.745929] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5707.746338] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5707.752349] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5707.757266] Lustre: Skipped 1 previous similar message [ 5711.019894] LustreError: 294467:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5711.063969] Lustre: server umount lustre-OST0000 complete [ 5712.865056] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5712.865775] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5712.869522] Lustre: Skipped 2 previous similar messages [ 5712.874213] Lustre: Skipped 1 previous similar message [ 5717.984192] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5717.986109] Lustre: Skipped 1 previous similar message [ 5718.527283] LustreError: 294668:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5718.529890] LustreError: 294668:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5718.561518] Lustre: server umount lustre-MDT0000 complete [ 5719.907671] LustreError: 293273:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764738918 with bad export cookie 7559699735513396735 [ 5719.912302] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5719.912450] LustreError: 293273:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5719.991438] LustreError: 294870:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5719.994352] LustreError: 294870:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5720.052959] Lustre: server umount lustre-MDT0001 complete [ 5722.875857] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 00:15:21 (1764738921) [ 5725.099806] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5725.280672] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5726.544215] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 5729.422745] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5729.544471] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5730.894366] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 5732.664854] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5733.889625] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5736.066685] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5738.020255] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 5739.694769] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5759.967812] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5759.970067] LustreError: Skipped 1 previous similar message [ 5759.971418] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5759.975323] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5760.993160] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5764.204105] LustreError: 297809:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5764.206720] LustreError: 297809:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5764.243339] Lustre: server umount lustre-OST0000 complete [ 5765.600092] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5765.602035] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5765.605392] Lustre: Skipped 1 previous similar message [ 5765.607515] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5765.609027] Lustre: Skipped 2 previous similar messages [ 5771.645653] LustreError: 298010:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5771.647741] LustreError: 298010:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5771.676340] Lustre: server umount lustre-MDT0000 complete [ 5772.970199] LustreError: 297159:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764738971 with bad export cookie 7559699735513397603 [ 5772.972076] LustreError: 296259:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000bd2:0x1:0x0]: rc = -5 [ 5772.973471] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5772.973895] LustreError: 297159:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5773.101202] Lustre: server umount lustre-MDT0001 complete [ 5776.639834] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 5777.705781] Key type lgssc unregistered [ 5777.846407] LNet: 298745:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5777.849689] LNetError: 298745:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5777.859453] LNet: Removed LNI 192.168.201.105@tcp [ 5778.167713] Key type .llcrypt unregistered [ 5778.168863] Key type ._llcrypt unregistered [ 5787.178364] Key type ._llcrypt registered [ 5787.180108] Key type .llcrypt registered [ 5787.222385] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [ 5793.510334] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 5793.872398] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5793.880809] alg: No test for adler32 (adler32-zlib) [ 5794.746232] Lustre: Lustre: Build Version: 2.17.0_RC1_3_g509e2bd [ 5794.836371] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 5796.415218] Key type lgssc registered [ 5796.836906] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5800.039801] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5802.338189] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5804.191543] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5806.116568] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5807.690573] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5809.296205] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5811.511300] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5811.532235] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5811.536423] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5812.614970] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5812.626061] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5812.659740] Lustre: lustre-MDT0000: new disk, initializing [ 5812.680842] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5812.686820] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5813.862916] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 5818.024744] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5818.055070] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5818.095056] Lustre: 302519:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5818.123231] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5818.125268] Lustre: Skipped 1 previous similar message [ 5818.160770] Lustre: lustre-MDT0001: new disk, initializing [ 5818.199612] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5818.212091] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5818.216090] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5819.549764] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 5822.542889] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5823.710206] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5825.860041] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5825.883640] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5825.979636] Lustre: lustre-OST2710: new disk, initializing [ 5825.981724] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 5826.011416] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 5827.720546] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 5827.896571] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 5827.900766] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 5827.937288] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 5830.704765] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 5832.868774] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5832.888572] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5832.945250] Lustre: lustre-OST03e8: new disk, initializing [ 5832.947787] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 5832.978814] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 5834.238301] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 5834.241723] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 5834.273770] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 5834.755429] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 5836.382855] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 5838.869071] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 5838.942222] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5840.195376] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 5840.267500] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5841.581531] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 5841.658693] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5842.908819] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 5842.975106] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5844.449631] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5844.452398] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5844.453814] Lustre: Skipped 3 previous similar messages [ 5844.455683] Lustre: Skipped 2 previous similar messages [ 5849.568534] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5849.571138] Lustre: Skipped 2 previous similar messages [ 5850.046597] LustreError: 306858:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5850.075318] Lustre: server umount lustre-MDT0000 complete [ 5851.271275] LustreError: 303301:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764739049 with bad export cookie 2085075306852907166 [ 5851.274268] LustreError: 303301:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5851.274515] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5851.343902] LustreError: 307059:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5851.345792] LustreError: 307059:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5851.395781] Lustre: server umount lustre-MDT0001 complete [ 5862.898464] LustreError: 307259:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5862.901325] LustreError: 307259:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5862.921041] Lustre: server umount lustre-OST2710 complete [ 5874.418980] LustreError: 307461:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 5874.421667] LustreError: 307461:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5874.466785] Lustre: server umount lustre-OST03e8 complete [ 5880.219455] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [ 5883.142371] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 5886.132516] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5888.134733] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5890.068307] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5892.099755] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5894.268717] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5894.289541] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5894.374490] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5894.385177] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5894.419200] Lustre: lustre-MDT0000: new disk, initializing [ 5894.442294] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5894.448308] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5895.705856] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 5899.759695] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5899.786917] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5899.819324] Lustre: 310253:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5899.935194] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5901.295346] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 5904.430245] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5905.678575] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5908.064644] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5908.092053] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5908.190805] Lustre: lustre-OST0000: new disk, initializing [ 5908.192406] Lustre: Skipped 1 previous similar message [ 5908.194128] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5908.196580] Lustre: Skipped 2 previous similar messages [ 5908.227392] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5908.230153] Lustre: Skipped 1 previous similar message [ 5910.201737] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5910.205103] Lustre: Skipped 1 previous similar message [ 5910.207554] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5910.243072] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5910.247580] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 5913.529667] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5915.014304] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5915.093696] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5916.491242] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5916.563399] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5920.740317] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5920.740702] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5920.742837] LustreError: Skipped 1 previous similar message [ 5920.743454] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5920.747994] Lustre: Skipped 1 previous similar message [ 5923.311097] LustreError: 313189:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5923.313604] LustreError: 313189:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5923.357491] Lustre: server umount lustre-OST0000 complete [ 5925.857091] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5925.857181] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5925.862094] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5925.866890] Lustre: Skipped 2 previous similar messages [ 5930.830929] Lustre: server umount lustre-MDT0000 complete [ 5932.093033] LustreError: 313393:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764739130 with bad export cookie 2085075306852908832 [ 5932.095837] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5932.096414] LustreError: 313393:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5932.161796] LustreError: 313595:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5932.164269] LustreError: 313595:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 5932.211775] Lustre: server umount lustre-MDT0001 complete [ 5935.101979] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 00:18:53 (1764739133) [ 5935.622387] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 5936.208371] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 00:18:54 (1764739134) [ 5937.220796] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 5939.949162] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5941.301236] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5942.643044] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5943.978617] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5944.380055] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5946.689421] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5946.783121] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5946.791977] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5946.853631] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5948.213797] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 5951.208027] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5951.245226] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5951.255952] Lustre: 315720:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5952.736163] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 5954.574413] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5955.758341] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5958.039639] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5958.068649] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5958.137019] Lustre: Denying initial registration attempt from nid 192.168.201.105@tcp, specified as failover [ 5958.140727] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 5958.144197] LustreError: 317334:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -99 [ 5958.147047] LustreError: 317334:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [ 5958.149666] LustreError: 317334:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 5958.174662] Lustre: server umount lustre-OST0000 complete [ 5958.176500] LustreError: 317334:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 5961.695781] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5961.698800] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5961.702940] Lustre: Skipped 1 previous similar message [ 5961.704957] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5966.593117] LustreError: 317703:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 5966.595094] LustreError: 317703:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5966.626614] Lustre: server umount lustre-MDT0000 complete [ 5967.990478] LustreError: 315712:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764739166 with bad export cookie 2085075306852909700 [ 5967.992580] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5967.995397] LustreError: 315712:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5968.112096] Lustre: server umount lustre-MDT0001 complete [ 5971.739795] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 5972.813345] Key type lgssc unregistered [ 5972.941748] LNet: 318437:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5972.945337] LNetError: 318437:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5972.955384] LNet: Removed LNI 192.168.201.105@tcp [ 5973.270546] Key type .llcrypt unregistered [ 5973.271726] Key type ._llcrypt unregistered [ 5977.031202] Key type ._llcrypt registered [ 5977.032676] Key type .llcrypt registered [ 5977.076857] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 5977.416979] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5977.470933] alg: No test for adler32 (adler32-zlib) [ 5978.336590] Lustre: Lustre: Build Version: 2.17.0_RC1_3_g509e2bd [ 5978.418136] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 5979.999166] Key type lgssc registered [ 5980.363344] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5982.761216] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5984.065473] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5985.283450] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5986.526525] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5986.863675] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5993.238964] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 5996.235948] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5996.242872] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5997.335848] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5997.344142] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5997.405599] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5998.864867] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 6001.835218] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6001.866170] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6001.877069] Lustre: 321170:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6001.971750] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6003.299194] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 6005.063777] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6006.226396] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6008.388254] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6008.419525] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6008.479269] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6008.534237] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6010.331932] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 6011.982386] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6012.649598] LustreError: 323443:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6012.683240] Lustre: server umount lustre-OST0000 complete [ 6013.922354] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6013.923783] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6013.927342] Lustre: Skipped 1 previous similar message [ 6014.447844] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6014.450128] Lustre: Skipped 2 previous similar messages [ 6019.551953] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6019.553789] Lustre: Skipped 1 previous similar message [ 6023.648158] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6023.650532] Lustre: Skipped 1 previous similar message [ 6028.255064] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6028.287097] LustreError: 323643:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 6028.289681] LustreError: 323643:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6028.313295] Lustre: server umount lustre-MDT0000 complete [ 6029.523221] LustreError: 321160:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764739227 with bad export cookie 8242217784568552878 [ 6029.526414] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6029.526831] LustreError: 321160:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6029.589517] LustreError: 323849:0:(obd_class.h:479:obd_check_dev()) Device 7 not setup [ 6029.592331] LustreError: 323849:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6029.644925] Lustre: server umount lustre-MDT0001 complete [ 6032.980650] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 6033.977377] Key type lgssc unregistered [ 6034.110392] LNet: 324380:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6034.114192] LNetError: 324380:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6034.123305] LNet: Removed LNI 192.168.201.105@tcp [ 6034.408758] Key type .llcrypt unregistered [ 6034.409938] Key type ._llcrypt unregistered [ 6043.012374] Key type ._llcrypt registered [ 6043.013748] Key type .llcrypt registered [ 6043.048252] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 6043.416096] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6043.429868] alg: No test for adler32 (adler32-zlib) [ 6044.284124] Lustre: Lustre: Build Version: 2.17.0_RC1_3_g509e2bd [ 6044.381963] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 6045.967186] Key type lgssc registered [ 6046.324364] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6048.916880] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6048.923472] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6050.065216] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6051.344938] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 6054.128910] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6054.251726] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6055.519097] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 6057.180924] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6058.367151] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6060.587042] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6060.701562] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6062.531770] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 6064.294141] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6085.087729] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6085.091523] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6085.097468] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6085.600854] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6085.601798] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6085.604373] Lustre: Skipped 1 previous similar message [ 6085.607235] Lustre: Skipped 2 previous similar messages [ 6088.128713] LustreError: 328023:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6088.159726] Lustre: server umount lustre-MDT0000 complete [ 6089.425539] LustreError: 325734:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764739287 with bad export cookie 6057593038219604417 [ 6089.429581] LustreError: 325734:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6089.430369] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6090.301940] LustreError: 325750:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6090.309140] LustreError: 325750:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6090.309822] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.5@tcp (stopping) [ 6090.720341] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6094.751860] LustreError: 327862:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6094.752221] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6094.758096] LustreError: 327862:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6094.760729] Lustre: Skipped 1 previous similar message [ 6095.549509] LustreError: 328226:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6095.552098] LustreError: 328226:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6095.598562] Lustre: server umount lustre-MDT0001 complete [ 6097.813422] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 6100.918686] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6114.271630] LustreError: 325063:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000003499f335 x1850463215444992/t0(0) o250->MGC192.168.201.105@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6114.394869] LustreError: 327362:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6114.401622] LustreError: 327362:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6114.422582] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6114.448414] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:161) [ 6115.796118] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 6118.790589] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6118.921732] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6119.973434] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6120.365774] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 6122.297718] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6124.071217] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6126.952588] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 4 sec [ 6128.176436] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6128.721449] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6130.144483] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6130.148581] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6130.153395] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6130.155391] Lustre: Skipped 1 previous similar message [ 6134.246238] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6134.250039] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6136.174113] LustreError: 330531:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6136.176921] LustreError: 330531:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6136.218045] Lustre: server umount lustre-OST0000 complete [ 6139.361387] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6139.361958] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6139.368225] Lustre: Skipped 3 previous similar messages [ 6143.683271] LustreError: 330732:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6143.686394] LustreError: 330732:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6143.722556] Lustre: server umount lustre-MDT0000 complete [ 6144.481370] LustreError: 329040:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6144.486306] LustreError: 329040:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6145.055376] LustreError: 325733:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764739343 with bad export cookie 6057593038219611578 [ 6145.058482] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6145.060912] LustreError: 325733:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6145.180549] Lustre: server umount lustre-MDT0001 complete [ 6148.859286] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 6149.902334] Key type lgssc unregistered [ 6150.032390] LNet: 331465:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6150.034968] LNetError: 331465:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6150.041415] LNet: Removed LNI 192.168.201.105@tcp [ 6150.332655] Key type .llcrypt unregistered [ 6150.333980] Key type ._llcrypt unregistered [ 6159.135411] Key type ._llcrypt registered [ 6159.136896] Key type .llcrypt registered [ 6159.175560] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 6159.554113] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6159.599850] alg: No test for adler32 (adler32-zlib) [ 6160.472528] Lustre: Lustre: Build Version: 2.17.0_RC1_3_g509e2bd [ 6160.564602] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 6162.151124] Key type lgssc registered [ 6162.549367] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6165.265400] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6165.272524] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6166.439153] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6167.778771] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 6170.744516] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6170.868050] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6172.251557] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 6174.055485] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6175.293629] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6181.345166] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6181.345292] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6181.346029] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6181.357306] Lustre: Skipped 1 previous similar message [ 6182.368393] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6182.370451] Lustre: Skipped 2 previous similar messages [ 6182.461592] LustreError: 334229:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6182.485545] Lustre: server umount lustre-MDT0000 complete [ 6183.788340] LustreError: 332819:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764739382 with bad export cookie 11789213319261629441 [ 6183.791248] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6183.793157] LustreError: 332819:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6183.859871] LustreError: 334433:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6183.862942] LustreError: 334433:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6183.911217] Lustre: server umount lustre-MDT0001 complete [ 6188.919362] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6189.024641] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6189.038107] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6189.126438] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6190.446243] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 6193.388846] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6193.422164] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6193.433976] Lustre: 335201:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6193.538892] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6194.811339] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 6196.416018] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6197.499304] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6199.909598] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6199.966505] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 6199.969392] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 6199.971882] LustreError: 336859:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -2 [ 6199.973909] LustreError: 336859:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [ 6199.975759] LustreError: 336859:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 6199.995864] Lustre: server umount lustre-OST0000 complete [ 6199.997033] LustreError: 336859:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 6202.608129] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6202.671713] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6202.743727] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6204.472437] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 6206.051544] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6208.026901] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:193) [ 6208.406594] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6208.433237] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6208.460715] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6208.491676] Lustre: lustre-OST0001: new disk, initializing [ 6208.493239] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6210.244189] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 6211.863625] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6212.776179] LustreError: 338967:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 6212.779016] LustreError: 338967:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6212.797038] Lustre: server umount lustre-OST0001 complete [ 6213.113419] LustreError: 337324:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6213.118460] LustreError: 337324:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6218.209413] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6218.210627] LustreError: 337326:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6218.211107] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6218.212069] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6218.212652] LustreError: Skipped 1 previous similar message [ 6218.216865] LustreError: 337326:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6220.011622] LustreError: 339177:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6220.013203] LustreError: 339177:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6220.046966] Lustre: server umount lustre-OST0000 complete [ 6223.328723] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6223.329548] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6223.332639] Lustre: Skipped 2 previous similar messages [ 6223.335491] Lustre: Skipped 2 previous similar messages [ 6227.326417] LustreError: 339379:0:(obd_class.h:479:obd_check_dev()) Device 24 not setup [ 6227.328276] LustreError: 339379:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6227.353370] Lustre: server umount lustre-MDT0000 complete [ 6228.448527] LustreError: 335939:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6228.455382] LustreError: 335939:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6228.585320] LustreError: 335924:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764739427 with bad export cookie 11789213319261629980 [ 6228.588881] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6228.588947] LustreError: 335924:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6228.701543] Lustre: server umount lustre-MDT0001 complete [ 6232.090461] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 6233.073221] Key type lgssc unregistered [ 6233.206416] LNet: 340111:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6233.209691] LNetError: 340111:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6233.219328] LNet: Removed LNI 192.168.201.105@tcp [ 6233.508917] Key type .llcrypt unregistered [ 6233.510081] Key type ._llcrypt unregistered [ 6244.174522] Key type ._llcrypt registered [ 6244.176010] Key type .llcrypt registered [ 6244.214249] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [ 6250.301297] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 6250.684526] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6250.727945] alg: No test for adler32 (adler32-zlib) [ 6251.596273] Lustre: Lustre: Build Version: 2.17.0_RC1_3_g509e2bd [ 6251.685895] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 6253.263184] Key type lgssc registered [ 6253.568594] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6256.528868] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6258.821093] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6260.942466] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6263.259897] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6265.665378] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6265.685457] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6265.691973] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6266.785114] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6266.800359] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6266.844367] Lustre: lustre-MDT0000: new disk, initializing [ 6266.872680] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6266.881297] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6268.280791] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 6272.245172] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6272.267373] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6272.298045] Lustre: 344031:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6272.321551] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6272.324562] Lustre: Skipped 1 previous similar message [ 6272.361972] Lustre: lustre-MDT0001: new disk, initializing [ 6272.394530] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6272.408269] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6272.412519] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6273.517469] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 6276.477092] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6277.647569] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6279.784983] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6279.806241] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6279.892074] Lustre: lustre-OST0000: new disk, initializing [ 6279.893780] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6279.920293] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6281.643971] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 6283.961094] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6283.964216] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6284.003455] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6284.601087] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6285.841324] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6285.912832] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6287.123459] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6287.190316] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6289.377819] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6289.378676] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6289.379277] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6289.379964] LustreError: Skipped 1 previous similar message [ 6289.383342] Lustre: Skipped 1 previous similar message [ 6293.804195] LustreError: 346967:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6293.839187] Lustre: server umount lustre-OST0000 complete [ 6298.080238] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6298.082905] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6298.087404] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6298.089712] Lustre: Skipped 1 previous similar message [ 6299.616659] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6299.617288] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6304.736219] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6304.739164] Lustre: Skipped 2 previous similar messages [ 6308.832824] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6308.835927] Lustre: Skipped 1 previous similar message [ 6309.343110] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6309.379630] LustreError: 347169:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6309.381546] LustreError: 347169:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6309.412903] Lustre: server umount lustre-MDT0000 complete [ 6310.723305] LustreError: 347170:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764739509 with bad export cookie 10299220795895094584 [ 6310.727023] LustreError: 347170:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6310.727141] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6310.806433] LustreError: 347372:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6310.809829] LustreError: 347372:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6310.860214] Lustre: server umount lustre-MDT0001 complete [ 6313.790417] Lustre: DEBUG MARKER: == conf-sanity test 60a: check mkfs.lustre --mkfsoptions -E -O options setting ========================================================== 00:25:11 (1764739511) [ 6315.832596] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6317.918721] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6325.603993] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [ 6328.729962] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 6331.861371] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6333.839476] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6335.754489] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6337.703282] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6339.806500] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6339.831491] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6339.905288] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6339.916788] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6339.949804] Lustre: lustre-MDT0000: new disk, initializing [ 6339.973402] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6339.981079] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6341.235539] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 6345.111480] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6345.137574] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6345.167791] Lustre: 351137:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6345.190877] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6345.193066] Lustre: Skipped 1 previous similar message [ 6345.228609] Lustre: lustre-MDT0001: new disk, initializing [ 6345.258796] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6345.269352] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6345.272477] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6346.644499] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 6349.723664] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6350.901525] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6353.136519] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6353.162409] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6353.255376] Lustre: lustre-OST0000: new disk, initializing [ 6353.257225] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6353.287657] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6354.616370] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6354.620484] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6354.656478] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6355.214814] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 6358.309936] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6359.716462] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6359.789209] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6361.155629] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6361.232132] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6365.153461] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6365.153740] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6365.157322] LustreError: Skipped 1 previous similar message [ 6365.164099] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6365.168146] Lustre: Skipped 1 previous similar message [ 6367.982052] LustreError: 354077:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6367.983775] LustreError: 354077:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6368.017297] Lustre: server umount lustre-OST0000 complete [ 6370.272757] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6370.276051] Lustre: Skipped 2 previous similar messages [ 6375.360096] LustreError: 354278:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6375.361936] LustreError: 354278:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6375.395218] Lustre: server umount lustre-MDT0000 complete [ 6376.610416] LustreError: 351128:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764739575 with bad export cookie 10299220795895095459 [ 6376.613144] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6376.615498] LustreError: 351128:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6376.737544] Lustre: server umount lustre-MDT0001 complete [ 6379.464610] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 00:26:17 (1764739577) [ 6381.902779] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 00:26:20 (1764739580) [ 6384.790497] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6384.962496] LustreError: 355494:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6384.970549] LustreError: 355494:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6384.990917] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6386.208370] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 6389.012121] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6390.434534] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 6392.301228] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6393.513335] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6395.820646] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6397.839769] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 6399.627645] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6413.735496] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 6414.322373] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 6414.905168] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 6415.502852] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 6416.181551] Lustre: Failing over lustre-MDT0000 [ 6416.267873] LustreError: 357955:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6416.270393] LustreError: 357955:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 6416.294881] Lustre: server umount lustre-MDT0000 complete [ 6419.935952] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6419.939188] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6419.943028] LustreError: 356204:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6425.568137] LustreError: 355494:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6425.572610] LustreError: 355494:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 6429.340110] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6429.391646] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6429.487986] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6429.491234] Lustre: Skipped 2 previous similar messages [ 6429.529116] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6430.782279] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6430.833537] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 6434.788962] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6434.795987] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 6434.810069] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6435.821605] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6436.310235] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6438.140649] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 6438.809264] Lustre: Failing over lustre-MDT0000 [ 6438.929087] LustreError: 359033:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6438.932772] LustreError: 359033:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6438.961764] Lustre: server umount lustre-MDT0000 complete [ 6439.906098] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6439.906149] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6439.906630] LustreError: 355488:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6439.906639] LustreError: 355488:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6439.921617] Lustre: Skipped 4 previous similar messages [ 6445.024855] LustreError: 355493:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6445.032034] LustreError: 355493:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 6451.918165] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6451.955498] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6452.066390] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6453.320037] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 6456.382874] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6457.318320] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6457.321694] Lustre: Skipped 2 previous similar messages [ 6457.328632] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 6457.349583] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 6458.521208] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6459.085914] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6462.432797] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6462.434611] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6462.437487] Lustre: Skipped 4 previous similar messages [ 6467.479187] Lustre: server umount lustre-MDT0000 complete [ 6467.553982] LustreError: 355493:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6467.562081] LustreError: 355493:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 6468.700913] LustreError: 357956:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764739667 with bad export cookie 10299220795895098126 [ 6468.704274] LustreError: 357956:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6468.704416] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6468.817888] Lustre: server umount lustre-MDT0001 complete [ 6480.243255] LustreError: 360466:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 6480.246057] LustreError: 360466:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 6480.291181] Lustre: server umount lustre-OST0000 complete [ 6485.567115] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6485.715093] LustreError: 361293:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6487.048842] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 6490.012163] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6491.448117] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 6493.290528] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6494.556838] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6496.879824] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6497.000404] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6497.003852] Lustre: Skipped 3 previous similar messages [ 6498.905209] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 6500.710435] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6511.340031] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 6520.800476] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6520.803089] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6520.806977] Lustre: Skipped 3 previous similar messages [ 6520.808740] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6523.597995] Lustre: server umount lustre-OST0000 complete [ 6539.231141] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6539.282706] Lustre: server umount lustre-MDT0000 complete [ 6540.511140] LustreError: 361272:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764739738 with bad export cookie 10299220795895098833 [ 6540.514601] LustreError: 361272:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6540.514618] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6540.630449] Lustre: server umount lustre-MDT0001 complete [ 6544.370198] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 6545.370659] Key type lgssc unregistered [ 6545.494404] LNet: 364495:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6545.496839] LNetError: 364495:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6545.508273] LNet: Removed LNI 192.168.201.105@tcp [ 6545.784091] Key type .llcrypt unregistered [ 6545.785289] Key type ._llcrypt unregistered [ 6555.569667] Key type ._llcrypt registered [ 6555.570841] Key type .llcrypt registered [ 6555.612204] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 6555.966028] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6556.031940] alg: No test for adler32 (adler32-zlib) [ 6556.887131] Lustre: Lustre: Build Version: 2.17.0_RC1_3_g509e2bd [ 6556.972923] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 6558.559145] Key type lgssc registered [ 6558.920629] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6561.534288] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6561.540666] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6562.702798] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6564.033204] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 6567.119586] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6567.263849] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6568.609622] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 6570.463207] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6571.732347] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6574.053350] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6574.171644] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6576.013800] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 6577.853574] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6588.523521] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 6598.112489] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6598.116649] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6598.122730] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6599.137348] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6599.137807] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6599.139879] Lustre: Skipped 1 previous similar message [ 6599.146024] Lustre: Skipped 1 previous similar message [ 6603.523553] LustreError: 368238:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6603.560286] Lustre: server umount lustre-MDT0000 complete [ 6604.256469] LustreError: 365963:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6604.264665] LustreError: 365963:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 6604.831186] LustreError: 365949:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764739803 with bad export cookie 9013196676854733830 [ 6604.834446] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6604.834489] LustreError: 365949:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6604.899596] LustreError: 368440:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6604.901676] LustreError: 368440:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6604.946898] Lustre: server umount lustre-MDT0001 complete [ 6616.437429] LustreError: 368641:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 6616.439658] LustreError: 368641:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6616.494654] Lustre: server umount lustre-OST0000 complete [ 6620.711448] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6620.846796] LustreError: 369370:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6620.867724] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6622.115766] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 6624.885833] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6624.996903] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6626.263156] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 6628.071465] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6629.257797] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6631.432892] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6633.569905] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 6635.566417] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6645.925149] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 6655.967933] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6655.971052] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6655.974564] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6655.977276] Lustre: Skipped 1 previous similar message [ 6659.131920] LustreError: 371740:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6659.133746] LustreError: 371740:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6659.159083] Lustre: server umount lustre-MDT0000 complete [ 6660.363945] LustreError: 369395:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764739858 with bad export cookie 9013196676854734929 [ 6660.366626] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6660.367872] LustreError: 369395:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6660.467318] Lustre: server umount lustre-MDT0001 complete [ 6671.987590] LustreError: 372143:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 6671.989303] LustreError: 372143:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 6672.033070] Lustre: server umount lustre-OST0000 complete [ 6677.616911] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6677.777309] LustreError: 372971:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6677.806889] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6677.809747] Lustre: Skipped 1 previous similar message [ 6679.265214] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 6682.206319] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6683.747524] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 6685.708405] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6686.998962] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6689.467560] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6691.537485] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 6693.361850] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6703.912234] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 6712.287827] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6712.290404] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6712.294697] Lustre: Skipped 2 previous similar messages [ 6712.296607] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6712.298628] Lustre: Skipped 3 previous similar messages [ 6714.339753] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6716.140213] LustreError: 375237:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6716.142397] LustreError: 375237:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6716.181354] Lustre: server umount lustre-OST0000 complete [ 6718.431917] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6718.434402] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6718.439181] Lustre: Skipped 1 previous similar message [ 6718.441082] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6718.443435] Lustre: Skipped 2 previous similar messages [ 6723.672764] Lustre: server umount lustre-MDT0000 complete [ 6724.576113] LustreError: 372971:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6724.582298] LustreError: 372971:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6724.962266] LustreError: 375239:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764739923 with bad export cookie 9013196676854735937 [ 6724.965469] LustreError: 375239:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6724.965508] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6725.088481] Lustre: server umount lustre-MDT0001 complete [ 6728.767548] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 6729.845253] Key type lgssc unregistered [ 6729.977337] LNet: 376174:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6729.979885] LNetError: 376174:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6729.990300] LNet: Removed LNI 192.168.201.105@tcp [ 6730.299654] Key type .llcrypt unregistered [ 6730.300746] Key type ._llcrypt unregistered [ 6739.774172] Key type ._llcrypt registered [ 6739.775840] Key type .llcrypt registered [ 6739.815633] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 6740.186975] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6740.228339] alg: No test for adler32 (adler32-zlib) [ 6741.082087] Lustre: Lustre: Build Version: 2.17.0_RC1_3_g509e2bd [ 6741.180721] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 6742.767228] Key type lgssc registered [ 6743.150536] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6745.907797] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6745.914952] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6745.918267] LustreError: 377572:0:(osd_handler.c:8561:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 6745.929058] LustreError: 377572:0:(obd_config.c:783:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 6745.931746] LustreError: 377572:0:(obd_mount.c:193:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 6745.935377] LustreError: 377572:0:(tgt_mount.c:2447:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 6745.939590] LustreError: 377572:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 6748.549668] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6748.552345] LustreError: 377969:0:(osd_handler.c:8561:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 6748.563922] LustreError: 377969:0:(obd_config.c:783:class_setup()) setup lustre-OST0000-osd failed (-22) [ 6748.566386] LustreError: 377969:0:(obd_mount.c:193:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 6748.570510] LustreError: 377969:0:(tgt_mount.c:2447:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 6748.575133] LustreError: 377969:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 6755.458053] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 6756.550458] Key type lgssc unregistered [ 6756.685413] LNet: 378953:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6756.687869] LNetError: 378953:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6756.700427] LNet: Removed LNI 192.168.201.105@tcp [ 6757.022572] Key type .llcrypt unregistered [ 6757.023746] Key type ._llcrypt unregistered [ 6763.271792] Key type ._llcrypt registered [ 6763.272799] Key type .llcrypt registered [ 6763.318644] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [ 6769.798653] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 6770.175932] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6770.238315] alg: No test for adler32 (adler32-zlib) [ 6771.106337] Lustre: Lustre: Build Version: 2.17.0_RC1_3_g509e2bd [ 6771.199021] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 6772.791218] Key type lgssc registered [ 6773.165086] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6776.299384] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6778.497896] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6780.647871] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6782.761401] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6785.063802] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6785.082624] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6785.087349] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6786.174121] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6786.188693] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6786.228031] Lustre: lustre-MDT0000: new disk, initializing [ 6786.254551] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6786.261418] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6787.625995] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 6791.934459] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6791.962485] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6791.999517] Lustre: 382057:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6792.025973] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6792.027951] Lustre: Skipped 1 previous similar message [ 6792.063710] Lustre: lustre-MDT0001: new disk, initializing [ 6792.097458] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6792.111084] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6792.114554] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6793.511946] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 6796.732870] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6798.045126] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6800.486770] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6800.519394] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6800.630842] Lustre: lustre-OST0000: new disk, initializing [ 6800.633174] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6800.672552] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6802.554989] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6802.557914] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6802.595101] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6802.759227] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 6805.956521] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6807.351462] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6807.440742] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6808.864936] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6808.949202] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6813.156445] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6813.158282] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6813.158676] LustreError: Skipped 1 previous similar message [ 6813.163030] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6813.165552] Lustre: Skipped 1 previous similar message [ 6815.661222] LustreError: 384998:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6815.704174] Lustre: server umount lustre-OST0000 complete [ 6817.760250] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6817.763623] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6817.768679] Lustre: Skipped 1 previous similar message [ 6817.771055] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6823.166304] LustreError: 385199:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6823.168800] LustreError: 385199:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6823.192888] Lustre: server umount lustre-MDT0000 complete [ 6824.478271] LustreError: 382050:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764740022 with bad export cookie 13823728449967628188 [ 6824.481822] LustreError: 382050:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6824.481845] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6824.549712] LustreError: 385400:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6824.551806] LustreError: 385400:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6824.606140] Lustre: server umount lustre-MDT0001 complete [ 6827.412378] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 6827.936140] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 00:33:46 (1764740026) [ 6830.255737] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6830.426937] LustreError: 386119:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6830.432705] LustreError: 386119:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6830.451735] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6831.826263] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 6834.737862] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6834.869342] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6836.331578] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 6838.185473] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6839.443813] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6841.824420] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6843.864061] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 6845.772248] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6848.243525] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6848.268272] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6848.330794] Lustre: lustre-OST0001: new disk, initializing [ 6848.334190] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6848.363605] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6848.367045] Lustre: Skipped 1 previous similar message [ 6850.427360] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 6852.353795] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6852.358620] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6852.389708] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6853.602326] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6857.695985] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6857.698711] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6857.701757] Lustre: Skipped 1 previous similar message [ 6857.703429] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6857.705268] Lustre: Skipped 3 previous similar messages [ 6859.845707] Lustre: lustre-OST0001: Not available for connect from 192.168.201.5@tcp (stopping) [ 6861.614071] LustreError: 389318:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 6861.616270] LustreError: 389318:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6861.637796] Lustre: server umount lustre-OST0001 complete [ 6865.894112] LustreError: 387713:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6865.899660] LustreError: 387713:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6871.014311] LustreError: 389108:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6871.018894] LustreError: 389108:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6876.131698] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6876.132362] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6876.132943] LustreError: 387715:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6876.134178] LustreError: Skipped 2 previous similar messages [ 6876.137980] Lustre: Skipped 1 previous similar message [ 6876.138573] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6876.151324] Lustre: Skipped 3 previous similar messages [ 6879.789164] LustreError: 389541:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6879.792624] LustreError: 389541:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6879.830907] Lustre: server umount lustre-OST0000 complete [ 6881.248801] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6881.248974] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6881.252595] Lustre: Skipped 2 previous similar messages [ 6886.368360] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6886.371733] Lustre: Skipped 3 previous similar messages [ 6887.253681] Lustre: server umount lustre-MDT0000 complete [ 6888.557451] LustreError: 386099:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764740086 with bad export cookie 13823728449967629056 [ 6888.561117] LustreError: 386099:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6888.561229] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6894.720083] LustreError: 389945:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 6894.722393] LustreError: 389945:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 6894.783467] Lustre: server umount lustre-MDT0001 complete [ 6898.406928] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 6899.531512] Key type lgssc unregistered [ 6899.666447] LNet: 390479:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6899.668891] LNetError: 390479:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6899.675477] LNet: Removed LNI 192.168.201.105@tcp [ 6900.046534] Key type .llcrypt unregistered [ 6900.047837] Key type ._llcrypt unregistered [ 6911.388258] Key type ._llcrypt registered [ 6911.389299] Key type .llcrypt registered [ 6911.425632] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [ 6917.991457] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 6918.319251] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6918.371057] alg: No test for adler32 (adler32-zlib) [ 6919.228931] Lustre: Lustre: Build Version: 2.17.0_RC1_3_g509e2bd [ 6919.328259] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 6920.927167] Key type lgssc registered [ 6921.315345] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6924.362764] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6926.591344] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6928.616212] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6930.813390] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6933.312510] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6933.334911] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6933.340415] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6934.430156] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6934.442073] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6934.483317] Lustre: lustre-MDT0000: new disk, initializing [ 6934.509157] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6934.516408] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6935.958072] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 6940.308373] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6940.335454] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6940.370622] Lustre: 394400:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6940.406167] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 6940.407570] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6940.412887] Lustre: Skipped 1 previous similar message [ 6940.453652] Lustre: lustre-MDT0001: new disk, initializing [ 6940.491360] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6940.505296] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6940.509942] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6941.970826] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 6945.227711] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6946.456536] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6948.811684] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6948.834674] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6948.931584] Lustre: lustre-OST0000: new disk, initializing [ 6948.933617] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6948.966200] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6950.960124] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 6954.133151] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6955.562703] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6958.074437] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6958.079474] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6958.114789] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6958.701840] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 6960.059777] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6960.132707] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6963.170243] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6963.170767] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6963.171422] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6963.179057] Lustre: Skipped 1 previous similar message [ 6966.894616] LustreError: 397364:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6966.940696] Lustre: server umount lustre-OST0000 complete [ 6968.288525] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6968.289118] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6968.294528] Lustre: Skipped 1 previous similar message [ 6968.298595] Lustre: Skipped 2 previous similar messages [ 6973.408488] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6973.411303] Lustre: Skipped 1 previous similar message [ 6974.405119] LustreError: 397566:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6974.408334] LustreError: 397566:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6974.444203] Lustre: server umount lustre-MDT0000 complete [ 6975.768419] LustreError: 394392:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764740174 with bad export cookie 1328511647416244594 [ 6975.771583] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6975.772975] LustreError: 394392:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6975.846771] LustreError: 397767:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6975.849665] LustreError: 397767:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6975.911429] Lustre: server umount lustre-MDT0001 complete [ 6978.695459] Lustre: DEBUG MARKER: == conf-sanity test 65: re-create the lost last_rcvd file when server mount ========================================================== 00:36:16 (1764740176) [ 6981.831222] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 6984.731662] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6984.876597] Lustre: lustre-MDT0000: new disk, initializing [ 6984.907936] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6986.186457] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 6989.116135] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6989.237853] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6990.563982] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 6992.369913] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6993.602663] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6994.279464] LustreError: 399063:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 6994.282725] LustreError: 399063:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 6994.286291] LustreError: 399063:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 9, retries 0, failed: rc = -5 [ 6994.400603] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6994.400637] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6994.401241] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6994.407271] Lustre: Skipped 1 previous similar message [ 6994.409256] LustreError: Skipped 1 previous similar message [ 7000.378179] LustreError: 400375:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7000.381300] LustreError: 400375:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 7000.406200] Lustre: server umount lustre-MDT0000 complete [ 7001.629381] LustreError: 399012:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764740200 with bad export cookie 1328511647416245476 [ 7001.632641] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7001.633971] LustreError: 399012:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7001.745871] Lustre: server umount lustre-MDT0001 complete [ 7004.687855] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 00:36:42 (1764740202) [ 7006.929776] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7007.082181] LustreError: 401294:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7007.091622] LustreError: 401294:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7007.106346] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7008.449946] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7011.354955] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7012.828350] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7014.584809] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7015.884445] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7018.231348] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7020.349642] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7022.194529] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7035.713650] Lustre: Permanently deactivating lustre-OST0000 [ 7035.720388] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7042.547501] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7042.891331] LustreError: 404009:0:(mgs_llog.c:1938:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [ 7042.895019] LustreError: 404009:0:(mgs_llog.c:1992:mgs_replace_nids()) Only MGS is allowed to be started [ 7042.898147] LustreError: 404009:0:(mgs_handler.c:1168:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 7043.559824] LustreError: 404058:0:(mgs_llog.c:1992:mgs_replace_nids()) Only MGS is allowed to be started [ 7043.562876] LustreError: 404058:0:(mgs_handler.c:1168:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 7050.286128] LustreError: 404155:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7050.288393] LustreError: 404155:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 7050.331298] Lustre: server umount lustre-OST0000 complete [ 7051.246777] LustreError: 404309:0:(mgs_llog.c:1992:mgs_replace_nids()) Only MGS is allowed to be started [ 7051.249207] LustreError: 404309:0:(mgs_handler.c:1168:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 7052.256568] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7052.257287] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7052.261781] Lustre: Skipped 1 previous similar message [ 7052.264451] Lustre: Skipped 3 previous similar messages [ 7062.499622] LustreError: 404406:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7062.501853] LustreError: 404406:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7062.526592] Lustre: server umount lustre-MDT0000 complete [ 7063.834247] LustreError: 401318:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764740262 with bad export cookie 1328511647416246008 [ 7063.837137] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7063.837582] LustreError: 401318:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7074.319429] Lustre: server umount lustre-MDT0001 complete [ 7077.258754] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7078.632410] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7079.612310] LustreError: 405450:0:(mgs_handler.c:1168:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [ 7081.705504] Lustre: 405788:0:(mgs_llog.c:1699:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.201.105@tcp [ 7082.411466] LustreError: 405885:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 7082.413663] LustreError: 405885:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 7082.439210] Lustre: server umount MGS complete [ 7086.232306] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7086.390834] LustreError: 406514:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7086.396182] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7086.398674] Lustre: Skipped 1 previous similar message [ 7086.422050] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7086.424170] Lustre: Skipped 2 previous similar messages [ 7087.714143] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7090.636173] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7090.733891] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7092.027610] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7093.881516] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7095.150751] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7095.796762] Lustre: Permanently reactivating lustre-OST0000 [ 7095.802373] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7095.804517] Lustre: Skipped 2 previous similar messages [ 7123.423157] LustreError: 408164:0:(osp_dev.c:692:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [ 7126.863560] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7126.985745] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7126.987849] Lustre: Skipped 1 previous similar message [ 7128.910679] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7130.706188] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7143.394532] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7143.396868] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7143.401028] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7143.402960] Lustre: Skipped 5 previous similar messages [ 7144.815071] LustreError: 409992:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7144.817411] LustreError: 409992:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 7144.853142] Lustre: server umount lustre-OST0000 complete [ 7146.975822] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7146.979758] LustreError: Skipped 1 previous similar message [ 7148.246656] Lustre: server umount lustre-MDT0000 complete [ 7148.512122] LustreError: 408261:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7148.518088] LustreError: 408261:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7149.524524] LustreError: 406494:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764740347 with bad export cookie 1328511647416246939 [ 7149.526369] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7149.528215] LustreError: 406494:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7149.533654] Lustre: 406505:0:(mgc_request.c:1910:mgc_process_log()) MGC192.168.201.105@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 7149.655898] Lustre: server umount lustre-MDT0001 complete [ 7153.352410] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 7154.493841] Key type lgssc unregistered [ 7154.638414] LNet: 410924:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7154.641953] LNetError: 410924:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7154.648457] LNet: Removed LNI 192.168.201.105@tcp [ 7154.974235] Key type .llcrypt unregistered [ 7154.975248] Key type ._llcrypt unregistered [ 7160.981500] Key type ._llcrypt registered [ 7160.982566] Key type .llcrypt registered [ 7161.016849] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [ 7167.400102] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 7167.858920] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7167.869885] alg: No test for adler32 (adler32-zlib) [ 7168.727968] Lustre: Lustre: Build Version: 2.17.0_RC1_3_g509e2bd [ 7168.815901] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 7170.399206] Key type lgssc registered [ 7170.795902] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7173.917936] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7176.201277] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7178.277833] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7180.461276] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7182.751935] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7182.773219] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7182.780487] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7183.873505] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7183.887860] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7183.926108] Lustre: lustre-MDT0000: new disk, initializing [ 7183.951405] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7183.958964] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7185.278307] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7189.504102] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7189.529365] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7189.570672] Lustre: 414030:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7189.597919] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7189.600056] Lustre: Skipped 1 previous similar message [ 7189.637365] Lustre: lustre-MDT0001: new disk, initializing [ 7189.681643] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7189.699411] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7189.703742] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7191.043742] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7194.198243] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7195.504388] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7197.878849] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7197.908154] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7198.008847] Lustre: lustre-OST0000: new disk, initializing [ 7198.011242] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7198.039142] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7200.060416] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7203.219389] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7204.348440] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7204.353235] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7204.392397] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7204.713309] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7204.792226] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7206.166721] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7206.243431] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7209.443473] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7209.445619] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7209.446033] LustreError: Skipped 1 previous similar message [ 7209.446528] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7209.449631] Lustre: Skipped 1 previous similar message [ 7209.453591] Lustre: Skipped 1 previous similar message [ 7212.975122] LustreError: 416969:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7213.021064] Lustre: server umount lustre-OST0000 complete [ 7214.560356] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7214.560920] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7214.563774] Lustre: Skipped 1 previous similar message [ 7219.680115] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7219.682703] Lustre: Skipped 2 previous similar messages [ 7220.414231] LustreError: 417170:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 7220.417374] LustreError: 417170:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7220.450769] Lustre: server umount lustre-MDT0000 complete [ 7221.722293] LustreError: 414807:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764740420 with bad export cookie 10687698032150191175 [ 7221.725574] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7221.727936] LustreError: 414807:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7221.797031] LustreError: 417371:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7221.798722] LustreError: 417371:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7221.847250] Lustre: server umount lustre-MDT0001 complete [ 7224.823639] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [ 7225.348741] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 00:40:23 (1764740423) [ 7227.604306] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7227.786083] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7229.150601] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7232.008800] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7232.126949] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7233.475994] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7235.279223] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7236.399988] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7238.611697] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7240.735945] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7242.504821] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7243.148911] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [ 7259.268430] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [ 7262.687994] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7262.691026] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7262.695503] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7264.736532] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7264.740542] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7266.861102] LustreError: 420547:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7266.863061] LustreError: 420547:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 7266.900175] Lustre: server umount lustre-OST0000 complete [ 7268.320113] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7268.322559] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7268.325548] Lustre: Skipped 2 previous similar messages [ 7269.857500] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7269.863489] Lustre: Skipped 1 previous similar message [ 7274.237205] LustreError: 420749:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7274.239427] LustreError: 420749:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7274.263734] Lustre: server umount lustre-MDT0000 complete [ 7274.975950] LustreError: 418088:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7274.981652] LustreError: 418088:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7275.494528] LustreError: 418070:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764740473 with bad export cookie 10687698032150192050 [ 7275.498323] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7275.501102] LustreError: 418070:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7275.614741] Lustre: server umount lustre-MDT0001 complete [ 7279.095577] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 7280.170208] Key type lgssc unregistered [ 7280.301423] LNet: 421483:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7280.305630] LNetError: 421483:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7280.314377] LNet: Removed LNI 192.168.201.105@tcp [ 7280.633149] Key type .llcrypt unregistered [ 7280.634391] Key type ._llcrypt unregistered [ 7288.593750] Key type ._llcrypt registered [ 7288.594982] Key type .llcrypt registered [ 7288.632698] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 7288.736320] Key type .llcrypt unregistered [ 7288.737646] Key type ._llcrypt unregistered [ 7295.605260] Key type ._llcrypt registered [ 7295.606347] Key type .llcrypt registered [ 7295.648428] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 7295.985889] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7295.993952] alg: No test for adler32 (adler32-zlib) [ 7296.854018] Lustre: Lustre: Build Version: 2.17.0_RC1_3_g509e2bd [ 7296.943875] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 7298.527144] Key type lgssc registered [ 7298.877804] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7301.391779] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7301.397358] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7302.550875] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7303.817265] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7306.618956] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7306.747826] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7308.744714] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7310.603321] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7311.842042] LustreError: 423627:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7311.849246] LustreError: 423627:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 7312.864359] LustreError: 423631:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7312.868706] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 7312.871951] LustreError: 423631:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7312.964990] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7313.098452] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7314.547302] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7323.103994] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7323.107024] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7323.110882] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7323.618786] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7323.619955] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7323.627653] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7327.918126] LustreError: 425685:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 7327.953073] Lustre: server umount lustre-OST0000 complete [ 7332.833615] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 7332.837539] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7332.844288] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7335.355534] LustreError: 425887:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7335.357280] LustreError: 425887:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7335.379518] Lustre: server umount lustre-MDT0000 complete [ 7336.630069] LustreError: 423613:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764740535 with bad export cookie 2178544690412757962 [ 7336.633515] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7336.633895] LustreError: 423613:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7336.699892] LustreError: 426089:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7336.702934] LustreError: 426089:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7336.752799] Lustre: server umount lustre-MDT0001 complete [ 7340.233282] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 7341.173594] Key type lgssc unregistered [ 7341.297419] LNet: 426620:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7341.299956] LNetError: 426620:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7341.307323] LNet: Removed LNI 192.168.201.105@tcp [ 7341.588745] Key type .llcrypt unregistered [ 7341.589926] Key type ._llcrypt unregistered [ 7350.832533] Key type ._llcrypt registered [ 7350.833544] Key type .llcrypt registered [ 7350.871166] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 7351.244563] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7351.286827] alg: No test for adler32 (adler32-zlib) [ 7352.151594] Lustre: Lustre: Build Version: 2.17.0_RC1_3_g509e2bd [ 7352.235961] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 7353.815178] Key type lgssc registered [ 7354.183554] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7356.837925] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7356.846255] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7372.319223] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7377.439250] LustreError: 427967:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.105@tcp: failed processing log, type 4: rc = -110 [ 7404.063173] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7405.986509] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7407.794024] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7410.041182] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7410.212659] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7410.232992] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 7411.520530] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7414.398469] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7414.545812] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7415.884259] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7417.684093] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7418.910246] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7423.970972] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7423.976509] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7423.978716] Lustre: Skipped 1 previous similar message [ 7427.309748] LustreError: 430249:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 7427.347521] Lustre: server umount lustre-OST0000 complete [ 7429.088727] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7429.092128] Lustre: Skipped 1 previous similar message [ 7429.094251] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7429.097592] Lustre: Skipped 1 previous similar message [ 7434.208240] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7434.210179] Lustre: Skipped 1 previous similar message [ 7434.748427] LustreError: 430450:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 7434.750452] LustreError: 430450:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7434.778519] Lustre: server umount lustre-MDT0000 complete [ 7436.026274] LustreError: 427974:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764740634 with bad export cookie 9578434127896193839 [ 7436.029517] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7436.030434] LustreError: 427974:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7436.099823] LustreError: 430652:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7436.102174] LustreError: 430652:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7436.158013] Lustre: server umount lustre-MDT0001 complete [ 7439.764331] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 7440.902533] Key type lgssc unregistered [ 7441.040356] LNet: 431183:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7441.044220] LNetError: 431183:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7441.053550] LNet: Removed LNI 192.168.201.105@tcp [ 7441.354146] Key type .llcrypt unregistered [ 7441.355254] Key type ._llcrypt unregistered [ 7450.731466] Key type ._llcrypt registered [ 7450.732506] Key type .llcrypt registered [ 7450.770262] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 7451.196776] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7451.231424] alg: No test for adler32 (adler32-zlib) [ 7452.096238] Lustre: Lustre: Build Version: 2.17.0_RC1_3_g509e2bd [ 7452.192497] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 7453.783130] Key type lgssc registered [ 7454.163460] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7456.859737] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7456.867267] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7458.031417] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7459.369310] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7462.325402] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7462.450288] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7463.858738] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7465.760642] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7467.050636] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7469.402290] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7469.518305] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7471.525251] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7473.503364] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7482.854383] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 7487.969804] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7487.971345] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7487.974124] Lustre: Skipped 2 previous similar messages [ 7487.974329] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7487.977086] Lustre: Skipped 2 previous similar messages [ 7489.095631] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.5@tcp (stopping) [ 7490.941450] LustreError: 434827:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7490.972539] Lustre: server umount lustre-MDT0000 complete [ 7491.741409] LustreError: 432543:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 192.168.201.5@tcp arrived at 1764740690 with bad export cookie 777713210877635617 [ 7491.747261] LustreError: 432543:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7492.403441] LustreError: 432543:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764740690 with bad export cookie 777713210877634994 [ 7492.408996] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7492.409275] LustreError: 432543:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7493.088359] LustreError: 433275:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7493.088870] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7493.092370] LustreError: 433275:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7493.102708] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7497.119852] LustreError: 433275:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7497.120139] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7497.125835] LustreError: 433275:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7498.478140] LustreError: 435030:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7498.481244] LustreError: 435030:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7498.519787] Lustre: server umount lustre-OST0000 complete [ 7500.769499] LustreError: 435384:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 7500.771579] LustreError: 435384:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7500.827922] Lustre: server umount lustre-MDT0001 complete [ 7504.438690] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 7505.460633] Key type lgssc unregistered [ 7505.578428] LNet: 435915:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7505.582054] LNetError: 435915:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7505.588316] LNet: Removed LNI 192.168.201.105@tcp [ 7505.889790] Key type .llcrypt unregistered [ 7505.890931] Key type ._llcrypt unregistered [ 7515.364607] Key type ._llcrypt registered [ 7515.365722] Key type .llcrypt registered [ 7515.409465] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 7515.818284] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7515.881433] alg: No test for adler32 (adler32-zlib) [ 7516.742456] Lustre: Lustre: Build Version: 2.17.0_RC1_3_g509e2bd [ 7516.827306] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 7518.415114] Key type lgssc registered [ 7518.759487] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7521.369266] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7521.376991] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7522.536968] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7523.829389] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7526.698870] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7526.833638] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7528.140988] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7529.945395] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7531.182532] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7533.412861] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7533.528813] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7535.371153] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7537.135127] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7546.856645] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 7551.970444] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7551.976777] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7551.978696] Lustre: Skipped 1 previous similar message [ 7552.583466] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.5@tcp (stopping) [ 7554.492535] LustreError: 439555:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 7554.514031] Lustre: server umount lustre-MDT0001 complete [ 7557.088167] LustreError: 438003:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7557.089454] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7557.089533] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7557.089540] Lustre: Skipped 1 previous similar message [ 7557.090053] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7557.094630] LustreError: 438003:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7562.208809] LustreError: 438003:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7562.209597] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7562.348095] LustreError: 439760:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7562.350187] LustreError: 439760:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7562.391775] Lustre: server umount lustre-OST0000 complete [ 7563.763423] LustreError: 439962:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7563.765457] LustreError: 439962:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7563.833282] Lustre: server umount lustre-MDT0000 complete [ 7568.794502] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 7569.857621] Key type lgssc unregistered [ 7569.999497] LNet: 440645:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7570.002485] LNetError: 440645:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7570.009652] LNet: Removed LNI 192.168.201.105@tcp [ 7570.357341] Key type .llcrypt unregistered [ 7570.358986] Key type ._llcrypt unregistered [ 7578.926598] Key type ._llcrypt registered [ 7578.927638] Key type .llcrypt registered [ 7578.965303] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [ 7585.372536] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 7585.726585] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7585.764974] alg: No test for adler32 (adler32-zlib) [ 7586.626164] Lustre: Lustre: Build Version: 2.17.0_RC1_3_g509e2bd [ 7586.724726] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 7588.303156] Key type lgssc registered [ 7588.663821] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7591.567348] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7593.714299] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7595.823706] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7597.877694] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7600.852049] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 7603.727715] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7605.940193] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7605.962185] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7605.966792] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7607.056067] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7607.070973] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7607.116958] Lustre: lustre-MDT0000: new disk, initializing [ 7607.142893] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7607.149794] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7608.430204] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7611.996668] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7614.173919] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7614.197547] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7614.285102] Lustre: lustre-OST0000: new disk, initializing [ 7614.287676] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7614.290092] Lustre: Skipped 1 previous similar message [ 7614.311760] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7616.102453] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7616.955483] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 7616.958566] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 7616.966908] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 7619.135884] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7622.260111] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7624.444421] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7624.472778] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7624.508539] Lustre: 444589:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7624.540606] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7624.575111] Lustre: lustre-MDT0001: new disk, initializing [ 7624.629789] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7624.648353] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 7624.653509] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 7625.973949] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7629.550281] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7629.625447] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7630.858944] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7636.027357] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 5 sec [ 7640.032505] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 7640.033881] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7640.036045] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7640.038551] Lustre: Skipped 1 previous similar message [ 7640.645066] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.5@tcp (stopping) [ 7640.647216] Lustre: Skipped 1 previous similar message [ 7643.775681] LustreError: 448044:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7643.799940] Lustre: server umount lustre-MDT0001 complete [ 7650.273453] LustreError: 447527:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7650.281111] LustreError: 447527:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 7655.393332] LustreError: 446811:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7655.397350] LustreError: 446811:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7657.951954] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7657.954315] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7657.958278] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7660.512419] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7660.513178] LustreError: 446811:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7662.124091] LustreError: 448316:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 7662.126356] LustreError: 448316:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7662.169850] Lustre: server umount lustre-OST0000 complete [ 7669.631994] LustreError: 448517:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 7669.634051] LustreError: 448517:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7669.694059] Lustre: server umount lustre-MDT0000 complete [ 7674.265494] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 7675.320689] Key type lgssc unregistered [ 7675.452401] LNet: 449202:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7675.454588] LNetError: 449202:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7675.461423] LNet: Removed LNI 192.168.201.105@tcp [ 7675.754069] Key type .llcrypt unregistered [ 7675.755122] Key type ._llcrypt unregistered [ 7690.143983] Key type ._llcrypt registered [ 7690.145515] Key type .llcrypt registered [ 7690.212650] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 7690.427366] Key type .llcrypt unregistered [ 7690.428980] Key type ._llcrypt unregistered [ 7698.547816] Key type ._llcrypt registered [ 7698.550287] Key type .llcrypt registered [ 7698.614936] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 7699.046039] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7699.071053] alg: No test for adler32 (adler32-zlib) [ 7700.017486] Lustre: Lustre: Build Version: 2.17.0_RC1_3_g509e2bd [ 7700.154752] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 7701.775264] Key type lgssc registered [ 7702.292585] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7705.927838] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7708.936213] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7711.956283] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7715.354639] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7715.384658] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7715.394745] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7716.524294] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7716.550213] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7716.591867] Lustre: lustre-MDT0000: new disk, initializing [ 7716.632960] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7716.643492] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7718.537657] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7723.292313] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7723.322653] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7723.367099] Lustre: 452536:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7723.398164] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7723.400661] Lustre: Skipped 1 previous similar message [ 7723.437759] Lustre: lustre-MDT0001: new disk, initializing [ 7723.477593] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7723.491517] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7723.494360] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7724.868904] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7728.068272] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7729.425110] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7731.902964] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7731.932710] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7732.061743] Lustre: lustre-OST0000: new disk, initializing [ 7732.064925] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7732.101349] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7734.243080] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7735.736933] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7735.740194] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7735.773308] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7737.488740] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7740.896865] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7740.898365] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7740.901807] Lustre: Skipped 2 previous similar messages [ 7740.903143] Lustre: Skipped 1 previous similar message [ 7745.856793] LustreError: 454980:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 7745.886426] Lustre: server umount lustre-MDT0000 complete [ 7747.200287] LustreError: 454894:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764740945 with bad export cookie 10652635752602906990 [ 7747.202689] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7747.204034] LustreError: 454894:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7747.273572] LustreError: 455182:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7747.275596] LustreError: 455182:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7747.328220] Lustre: server umount lustre-MDT0001 complete [ 7758.834765] LustreError: 455382:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 7758.837647] LustreError: 455382:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7758.890053] Lustre: server umount lustre-OST0000 complete [ 7762.664293] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 00:49:20 (1764740960) [ 7763.029414] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7765.412585] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7765.582084] LustreError: 456208:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7765.589581] LustreError: 456208:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 7765.606572] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7766.958230] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7769.844048] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7769.970833] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7771.329191] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7773.311429] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7774.534972] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7776.810049] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7776.834443] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7776.900442] Lustre: Found index 0 for lustre-OST0000, updating log [ 7778.940843] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7780.699979] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7791.333444] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [ 7800.800707] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7800.802698] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7800.807494] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7800.808993] Lustre: Skipped 1 previous similar message [ 7801.824684] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7801.827550] Lustre: Skipped 2 previous similar messages [ 7803.500127] LustreError: 458489:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7803.501917] LustreError: 458489:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7803.536255] Lustre: server umount lustre-OST0000 complete [ 7804.461834] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7805.919788] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7805.921685] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7805.925218] Lustre: Skipped 1 previous similar message [ 7805.927108] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7811.200041] LustreError: 458743:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7811.202253] LustreError: 458743:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7811.232163] Lustre: server umount lustre-MDT0000 complete [ 7812.591162] LustreError: 456189:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764741011 with bad export cookie 10652635752602908369 [ 7812.595125] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7812.595234] LustreError: 456189:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7812.718620] Lustre: server umount lustre-MDT0001 complete [ 7815.569732] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 00:50:13 (1764741013) [ 7815.910683] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7816.262302] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7816.650763] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7817.002781] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7817.347756] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7817.691228] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7818.059639] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7818.439672] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7818.827899] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7819.172132] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7819.541912] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7819.916252] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7820.300432] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7820.674569] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7821.037847] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7821.403672] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7821.743484] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7822.106944] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7822.472194] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7822.846434] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7823.182454] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7823.529231] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7823.874275] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7824.223928] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7824.572187] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7824.928226] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7825.290478] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7825.629798] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7825.985885] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7826.349481] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7826.701888] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7827.026783] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7827.340339] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7827.661551] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7827.975689] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7828.324750] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7828.663467] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7829.022168] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7829.352388] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7829.697501] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7830.010971] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7830.334633] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7830.673919] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7831.019988] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7831.371457] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7831.715500] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7832.047705] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7832.372902] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7832.701469] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7833.037543] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7833.378094] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7835.880450] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7836.039111] LustreError: 462329:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7836.066078] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7836.068846] Lustre: Skipped 1 previous similar message [ 7836.383271] LNet: 1 local NIs in recovery (showing 1): 192.168.201.105@tcp [ 7837.387780] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7840.227084] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7841.708262] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7843.598704] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7844.830118] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7847.161772] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7847.197848] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7847.293769] Lustre: Found index 0 for lustre-OST0000, updating log [ 7848.429549] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:65) [ 7849.463200] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7851.550235] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7853.535954] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7853.539642] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7853.544630] Lustre: Skipped 1 previous similar message [ 7853.546646] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7853.549359] Lustre: Skipped 4 previous similar messages [ 7858.290080] LustreError: 464622:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7858.292252] LustreError: 464622:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 7858.333477] Lustre: server umount lustre-OST0000 complete [ 7860.704526] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7860.709442] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7860.715245] Lustre: Skipped 1 previous similar message [ 7861.732728] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7861.736360] Lustre: Skipped 3 previous similar messages [ 7865.951855] Lustre: server umount lustre-MDT0000 complete [ 7866.848276] LustreError: 462329:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7866.856891] LustreError: 462329:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7867.429248] LustreError: 463993:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764741065 with bad export cookie 10652635752602909440 [ 7867.432185] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7867.434121] LustreError: 463993:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7867.558862] Lustre: server umount lustre-MDT0001 complete [ 7868.614846] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7874.539967] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [ 7878.147826] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 7882.081237] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7884.663256] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7886.971444] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7889.431769] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7892.105255] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 00:51:30 (1764741090) [ 7897.823074] LNet: 1 local NIs in recovery (showing 1): 192.168.201.105@tcp [ 7915.033419] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 7916.077402] Key type lgssc unregistered [ 7916.210431] LNet: 468615:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7916.212866] LNetError: 468615:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8046.310343] LNet: Removed LNI 192.168.201.105@tcp [ 8046.597821] Key type .llcrypt unregistered [ 8046.598882] Key type ._llcrypt unregistered [ 8053.071905] Key type ._llcrypt registered [ 8053.073344] Key type .llcrypt registered [ 8053.113184] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 8053.490939] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8053.525542] alg: No test for adler32 (adler32-zlib) [ 8054.374974] Lustre: Lustre: Build Version: 2.17.0_RC1_3_g509e2bd [ 8054.460446] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 8056.039928] Key type lgssc registered [ 8056.384082] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8057.793207] LNet: Added LNI 192.168.201.105@tcp100 [8/256/0/180] [ 8058.069644] LNet: Added LNI 192.168.201.105@tcp101 [8/256/0/180] [ 8058.343815] LNet: Added LNI 192.168.201.105@tcp102 [8/256/0/180] [ 8058.638408] LNet: Added LNI 192.168.201.105@tcp103 [8/256/0/180] [ 8058.942117] LNet: Added LNI 192.168.201.105@tcp104 [8/256/0/180] [ 8059.247191] LNet: Added LNI 192.168.201.105@tcp105 [8/256/0/180] [ 8059.548078] LNet: Added LNI 192.168.201.105@tcp106 [8/256/0/180] [ 8059.847149] LNet: Added LNI 192.168.201.105@tcp107 [8/256/0/180] [ 8060.147639] LNet: Added LNI 192.168.201.105@tcp108 [8/256/0/180] [ 8060.451367] LNet: Added LNI 192.168.201.105@tcp109 [8/256/0/180] [ 8060.763371] LNet: Added LNI 192.168.201.105@tcp110 [8/256/0/180] [ 8061.068485] LNet: Added LNI 192.168.201.105@tcp111 [8/256/0/180] [ 8061.361316] LNet: Added LNI 192.168.201.105@tcp112 [8/256/0/180] [ 8061.678527] LNet: Added LNI 192.168.201.105@tcp113 [8/256/0/180] [ 8061.979245] LNet: Added LNI 192.168.201.105@tcp114 [8/256/0/180] [ 8062.270545] LNet: Added LNI 192.168.201.105@tcp115 [8/256/0/180] [ 8062.580640] LNet: Added LNI 192.168.201.105@tcp116 [8/256/0/180] [ 8062.886459] LNet: Added LNI 192.168.201.105@tcp117 [8/256/0/180] [ 8063.170281] LNet: Added LNI 192.168.201.105@tcp118 [8/256/0/180] [ 8063.455541] LNet: Added LNI 192.168.201.105@tcp119 [8/256/0/180] [ 8063.748688] LNet: Added LNI 192.168.201.105@tcp120 [8/256/0/180] [ 8064.047298] LNet: Added LNI 192.168.201.105@tcp121 [8/256/0/180] [ 8064.336844] LNet: Added LNI 192.168.201.105@tcp122 [8/256/0/180] [ 8064.627225] LNet: Added LNI 192.168.201.105@tcp123 [8/256/0/180] [ 8064.907320] LNet: Added LNI 192.168.201.105@tcp124 [8/256/0/180] [ 8065.197491] LNet: Added LNI 192.168.201.105@tcp125 [8/256/0/180] [ 8065.489341] LNet: Added LNI 192.168.201.105@tcp126 [8/256/0/180] [ 8065.778269] LNet: Added LNI 192.168.201.105@tcp127 [8/256/0/180] [ 8066.058099] LNet: Added LNI 192.168.201.105@tcp128 [8/256/0/180] [ 8066.340649] LNet: Added LNI 192.168.201.105@tcp129 [8/256/0/180] [ 8066.622084] LNet: Added LNI 192.168.201.105@tcp130 [8/256/0/180] [ 8066.918358] LNet: Added LNI 192.168.201.105@tcp131 [8/256/0/180] [ 8067.218756] LNet: Added LNI 192.168.201.105@tcp132 [8/256/0/180] [ 8067.505615] LNet: Added LNI 192.168.201.105@tcp133 [8/256/0/180] [ 8067.793309] LNet: Added LNI 192.168.201.105@tcp134 [8/256/0/180] [ 8068.079661] LNet: Added LNI 192.168.201.105@tcp135 [8/256/0/180] [ 8070.229041] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8070.252105] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8070.258570] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8071.296496] Lustre: 471605:0:(lustre_peer.c:105:class_add_uuid()) 192.168.201.105@tcp: can't add NID 192.168.201.105@tcp130: rc = -75 [ 8071.365853] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8071.379599] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8071.419373] Lustre: lustre-MDT0000: new disk, initializing [ 8071.448945] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8071.456275] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8072.353537] LNetError: 469066:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.105 [ 8072.357772] LNetError: Protocol error connecting to 192.168.201.105@tcp105 at host 192.168.201.105:988: is it running a compatible version of Lustre? [ 8072.687224] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 8076.762639] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8076.783678] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8076.823674] Lustre: 471623:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8076.848817] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8076.850897] Lustre: Skipped 1 previous similar message [ 8076.883720] Lustre: lustre-MDT0001: new disk, initializing [ 8076.895208] Lustre: 472413:0:(lustre_peer.c:105:class_add_uuid()) 192.168.201.105@tcp: can't add NID 192.168.201.105@tcp130: rc = -75 [ 8076.897740] Lustre: 472413:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [ 8076.935671] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8076.948817] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8076.952448] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8078.182669] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 8079.283602] Lustre: 472761:0:(lustre_peer.c:105:class_add_uuid()) 192.168.201.105@tcp: can't add NID 192.168.201.105@tcp130: rc = -75 [ 8079.287712] Lustre: 472761:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [ 8081.130883] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8082.262752] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8084.649252] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8084.682921] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8084.792378] Lustre: lustre-OST0000: new disk, initializing [ 8084.794979] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8084.829622] Lustre: 473381:0:(lustre_peer.c:105:class_add_uuid()) 192.168.201.105@tcp: can't add NID 192.168.201.105@tcp130: rc = -75 [ 8084.833770] Lustre: 473381:0:(lustre_peer.c:105:class_add_uuid()) Skipped 5 previous similar messages [ 8084.854945] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8086.811180] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 8088.189699] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8088.194466] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8088.231368] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8089.890526] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8093.665192] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8093.666309] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8093.668151] LustreError: Skipped 1 previous similar message [ 8093.668802] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8093.671869] Lustre: Skipped 1 previous similar message [ 8094.319116] LustreError: 474216:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 8094.362981] Lustre: server umount lustre-OST0000 complete [ 8097.759959] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8097.763426] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8097.767404] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8097.769053] Lustre: Skipped 1 previous similar message [ 8098.784046] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8098.784494] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8101.760166] LustreError: 474416:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 8101.762886] LustreError: 474416:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8101.791294] Lustre: server umount lustre-MDT0000 complete [ 8103.044622] LustreError: 472412:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764741301 with bad export cookie 16201121663162203444 [ 8103.047784] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8103.048690] LustreError: 472412:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8103.115853] LustreError: 474617:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8103.117572] LustreError: 474617:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8103.174615] Lustre: server umount lustre-MDT0001 complete [ 8106.784401] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 8107.853591] Key type lgssc unregistered [ 8108.040341] LNet: 475156:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8108.044824] LNetError: 475156:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8108.047959] LNet: Removed LNI 192.168.201.105@tcp [ 8109.087207] LNet: Removed LNI 192.168.201.105@tcp100 [ 8109.089178] LNetError: 475156:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8109.091725] LNetError: 475156:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 8109.095406] LNet: Removed LNI 192.168.201.105@tcp101 [ 8110.111154] LNet: Removed LNI 192.168.201.105@tcp102 [ 8110.112847] LNetError: 475156:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8110.114824] LNetError: 475156:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 8110.118033] LNet: Removed LNI 192.168.201.105@tcp103 [ 8111.135184] LNet: Removed LNI 192.168.201.105@tcp104 [ 8112.159145] LNet: Removed LNI 192.168.201.105@tcp105 [ 8112.161159] LNetError: 475156:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8112.164374] LNetError: 475156:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 8112.167784] LNet: Removed LNI 192.168.201.105@tcp106 [ 8113.183190] LNet: Removed LNI 192.168.201.105@tcp107 [ 8114.207180] LNet: Removed LNI 192.168.201.105@tcp108 [ 8115.231239] LNet: Removed LNI 192.168.201.105@tcp109 [ 8116.255137] LNet: Removed LNI 192.168.201.105@tcp110 [ 8116.256885] LNetError: 475156:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8116.259137] LNetError: 475156:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 8116.261308] LNet: Removed LNI 192.168.201.105@tcp111 [ 8117.279169] LNet: Removed LNI 192.168.201.105@tcp112 [ 8118.303117] LNet: Removed LNI 192.168.201.105@tcp113 [ 8119.327190] LNet: Removed LNI 192.168.201.105@tcp114 [ 8120.351229] LNet: Removed LNI 192.168.201.105@tcp115 [ 8121.375203] LNet: Removed LNI 192.168.201.105@tcp116 [ 8122.399157] LNet: Removed LNI 192.168.201.105@tcp117 [ 8122.400524] LNet: Removed LNI 192.168.201.105@tcp118 [ 8122.401582] LNet: Removed LNI 192.168.201.105@tcp119 [ 8122.402880] LNet: Removed LNI 192.168.201.105@tcp120 [ 8122.404445] LNet: Removed LNI 192.168.201.105@tcp121 [ 8122.405533] LNet: Removed LNI 192.168.201.105@tcp122 [ 8122.406524] LNet: Removed LNI 192.168.201.105@tcp123 [ 8122.407718] LNet: Removed LNI 192.168.201.105@tcp124 [ 8122.408859] LNet: Removed LNI 192.168.201.105@tcp125 [ 8122.410034] LNet: Removed LNI 192.168.201.105@tcp126 [ 8122.411239] LNet: Removed LNI 192.168.201.105@tcp127 [ 8122.412305] LNet: Removed LNI 192.168.201.105@tcp128 [ 8122.413474] LNet: Removed LNI 192.168.201.105@tcp129 [ 8122.414555] LNet: Removed LNI 192.168.201.105@tcp130 [ 8122.415676] LNet: Removed LNI 192.168.201.105@tcp131 [ 8122.416870] LNet: Removed LNI 192.168.201.105@tcp132 [ 8122.418480] LNet: Removed LNI 192.168.201.105@tcp133 [ 8122.420036] LNet: Removed LNI 192.168.201.105@tcp134 [ 8122.427475] LNet: Removed LNI 192.168.201.105@tcp135 [ 8122.702256] Key type .llcrypt unregistered [ 8122.703777] Key type ._llcrypt unregistered [ 8130.087861] Key type ._llcrypt registered [ 8130.089495] Key type .llcrypt registered [ 8130.130018] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [ 8136.275805] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 8136.633535] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8136.672073] alg: No test for adler32 (adler32-zlib) [ 8137.523910] Lustre: Lustre: Build Version: 2.17.0_RC1_3_g509e2bd [ 8137.612592] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 8139.199195] Key type lgssc registered [ 8139.570932] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8142.550271] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8144.853381] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8146.912194] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8149.114140] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8151.410952] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 00:55:49 (1764741349) [ 8157.299471] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 8158.324402] Key type lgssc unregistered [ 8158.454439] LNet: 479169:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8158.457041] LNetError: 479169:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8158.466384] LNet: Removed LNI 192.168.201.105@tcp [ 8158.775395] Key type .llcrypt unregistered [ 8158.777090] Key type ._llcrypt unregistered [ 8165.079132] Key type ._llcrypt registered [ 8165.080036] Key type .llcrypt registered [ 8165.115889] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 8165.434620] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8165.477949] alg: No test for adler32 (adler32-zlib) [ 8166.328821] Lustre: Lustre: Build Version: 2.17.0_RC1_3_g509e2bd [ 8166.412668] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 8167.999178] Key type lgssc registered [ 8168.366980] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8171.373839] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8171.393333] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8171.398452] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8172.475431] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8172.488796] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8172.519948] Lustre: lustre-MDT0000: new disk, initializing [ 8172.541523] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8172.548053] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8173.870491] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 8178.019545] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8178.044465] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8178.077181] Lustre: 480339:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8178.100656] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8178.102207] Lustre: Skipped 1 previous similar message [ 8178.133783] Lustre: lustre-MDT0001: new disk, initializing [ 8178.165537] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8178.178429] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8178.182408] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8179.454146] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 8182.536555] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8183.766825] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8185.956519] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8185.976926] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8186.070031] Lustre: lustre-OST0000: new disk, initializing [ 8186.072709] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8186.100759] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8188.021805] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 8191.037100] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8192.231717] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing lnet_if_list [ 8193.532690] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8193.536604] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8193.575749] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8193.846630] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [ 8193.860371] LNet: Added LNI 192.168.201.105@tcp10 [8/256/0/180] [ 8195.727968] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [ 8195.742452] LNet: Added LNI 192.168.201.105@tcp11 [8/256/0/180] [ 8197.037043] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [ 8197.048828] LNet: Added LNI 192.168.201.105@tcp12 [8/256/0/180] [ 8198.342592] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [ 8198.356232] LNet: Added LNI 192.168.201.105@tcp13 [8/256/0/180] [ 8199.804840] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [ 8199.818845] LNet: Added LNI 192.168.201.105@tcp14 [8/256/0/180] [ 8201.192956] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [ 8201.204889] LNet: Added LNI 192.168.201.105@tcp15 [8/256/0/180] [ 8202.602706] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [ 8202.615075] LNet: Added LNI 192.168.201.105@tcp16 [8/256/0/180] [ 8204.201360] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [ 8204.213545] LNet: Added LNI 192.168.201.105@tcp17 [8/256/0/180] [ 8205.709246] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [ 8205.718337] LNet: Added LNI 192.168.201.105@tcp18 [8/256/0/180] [ 8207.416167] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [ 8207.430326] LNet: Added LNI 192.168.201.105@tcp19 [8/256/0/180] [ 8209.021770] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [ 8209.037444] LNet: Added LNI 192.168.201.105@tcp20 [8/256/0/180] [ 8210.632682] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [ 8210.645825] LNet: Added LNI 192.168.201.105@tcp21 [8/256/0/180] [ 8212.331892] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [ 8212.345324] LNet: Added LNI 192.168.201.105@tcp22 [8/256/0/180] [ 8214.091307] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [ 8214.106537] LNet: Added LNI 192.168.201.105@tcp23 [8/256/0/180] [ 8215.878592] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [ 8215.890322] LNet: Added LNI 192.168.201.105@tcp24 [8/256/0/180] [ 8217.769829] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [ 8217.782067] LNet: Added LNI 192.168.201.105@tcp25 [8/256/0/180] [ 8219.680592] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [ 8219.692925] LNet: Added LNI 192.168.201.105@tcp26 [8/256/0/180] [ 8221.650562] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [ 8221.662048] LNet: Added LNI 192.168.201.105@tcp27 [8/256/0/180] [ 8221.668708] LNetError: 479602:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.105 [ 8221.674206] LNetError: Protocol error connecting to 192.168.201.105@tcp26 at host 192.168.201.105:988: is it running a compatible version of Lustre? [ 8223.527177] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [ 8223.538756] LNet: Added LNI 192.168.201.105@tcp28 [8/256/0/180] [ 8223.542149] LNetError: 479599:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.105 [ 8223.545724] LNetError: Protocol error connecting to 192.168.201.105@tcp26 at host 192.168.201.105:988: is it running a compatible version of Lustre? [ 8225.443047] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [ 8225.454436] LNet: Added LNI 192.168.201.105@tcp29 [8/256/0/180] [ 8225.458654] LNetError: 479600:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.105 [ 8225.461267] LNetError: Protocol error connecting to 192.168.201.105@tcp28 at host 192.168.201.105:988: is it running a compatible version of Lustre? [ 8225.823161] LNet: 1 local NIs in recovery (showing 1): 192.168.201.105@tcp28 [ 8227.363925] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [ 8227.372663] LNet: Added LNI 192.168.201.105@tcp30 [8/256/0/180] [ 8228.897514] LNetError: 479601:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.105 [ 8228.901885] LNetError: 479601:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 1 previous similar message [ 8228.904355] LNetError: Protocol error connecting to 192.168.201.105@tcp28 at host 192.168.201.105:988: is it running a compatible version of Lustre? [ 8228.908646] LNetError: Skipped 1 previous similar message [ 8229.436308] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [ 8229.449487] LNet: Added LNI 192.168.201.105@tcp31 [8/256/0/180] [ 8231.540026] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [ 8231.549830] LNet: Added LNI 192.168.201.105@tcp32 [8/256/0/180] [ 8232.996298] LNetError: 479601:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.105 [ 8233.000619] LNetError: 479601:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 3 previous similar messages [ 8233.003884] LNetError: Protocol error connecting to 192.168.201.105@tcp29 at host 192.168.201.105:988: is it running a compatible version of Lustre? [ 8233.009207] LNetError: Skipped 3 previous similar messages [ 8233.612842] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [ 8233.626419] LNet: Added LNI 192.168.201.105@tcp33 [8/256/0/180] [ 8235.862824] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [ 8235.875758] LNet: Added LNI 192.168.201.105@tcp34 [8/256/0/180] [ 8238.262062] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [ 8238.276598] LNet: Added LNI 192.168.201.105@tcp35 [8/256/0/180] [ 8240.547826] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [ 8240.561546] LNet: Added LNI 192.168.201.105@tcp36 [8/256/0/180] [ 8241.184360] LNetError: 479600:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.105 [ 8241.187274] LNetError: 479600:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 10 previous similar messages [ 8241.189780] LNetError: Protocol error connecting to 192.168.201.105@tcp35 at host 192.168.201.105:988: is it running a compatible version of Lustre? [ 8241.193387] LNetError: Skipped 10 previous similar messages [ 8242.931144] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [ 8242.944520] LNet: Added LNI 192.168.201.105@tcp37 [8/256/0/180] [ 8245.259602] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [ 8245.272856] LNet: Added LNI 192.168.201.105@tcp38 [8/256/0/180] [ 8246.252076] LNetError: 479609:0:(peer.c:1877:lnet_peer_add_nid()) Peer 192.168.201.105@tcp NID 192.168.201.105@tcp is already locked with peer 192.168.201.105@tcp [ 8246.258815] LNetError: 479609:0:(lib-move.c:2321:lnet_handle_find_routed_path()) peer 192.168.201.105@tcp has no available nets [ 8247.776970] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [ 8247.787824] LNet: Added LNI 192.168.201.105@tcp39 [8/256/0/180] [ 8247.789588] LNetError: 479609:0:(lib-move.c:2321:lnet_handle_find_routed_path()) peer 192.168.201.105@tcp has no available nets [ 8250.095362] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [ 8250.109605] LNet: Added LNI 192.168.201.105@tcp40 [8/256/0/180] [ 8250.111565] LNetError: 479609:0:(lib-move.c:2321:lnet_handle_find_routed_path()) peer 192.168.201.105@tcp has no available nets [ 8250.114254] LNetError: 479609:0:(lib-move.c:2321:lnet_handle_find_routed_path()) Skipped 1 previous similar message [ 8252.454608] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [ 8252.464924] LNet: Added LNI 192.168.201.105@tcp41 [8/256/0/180] [ 8254.801783] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [ 8254.813227] LNet: Added LNI 192.168.201.105@tcp42 [8/256/0/180] [ 8257.217696] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [ 8257.230737] LNet: Added LNI 192.168.201.105@tcp43 [8/256/0/180] [ 8257.569277] LNetError: 479599:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.105 [ 8257.574247] LNetError: 479599:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 26 previous similar messages [ 8257.578035] LNetError: Protocol error connecting to 192.168.201.105@tcp41 at host 192.168.201.105:988: is it running a compatible version of Lustre? [ 8257.582842] LNetError: Skipped 26 previous similar messages [ 8259.618934] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [ 8259.629734] LNet: Added LNI 192.168.201.105@tcp44 [8/256/0/180] [ 8262.274823] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [ 8262.286747] LNet: Added LNI 192.168.201.105@tcp45 [8/256/0/180] [ 8265.037522] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [ 8265.051108] LNet: Added LNI 192.168.201.105@tcp46 [8/256/0/180] [ 8267.736775] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [ 8267.749851] LNet: Added LNI 192.168.201.105@tcp47 [8/256/0/180] [ 8270.518539] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [ 8270.531572] LNet: Added LNI 192.168.201.105@tcp48 [8/256/0/180] [ 8273.333700] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [ 8273.346336] LNet: Added LNI 192.168.201.105@tcp49 [8/256/0/180] [ 8276.349392] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [ 8276.362937] LNet: Added LNI 192.168.201.105@tcp50 [8/256/0/180] [ 8285.666180] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8285.667492] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8285.668960] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8285.671497] Lustre: Skipped 1 previous similar message [ 8285.674158] Lustre: Skipped 1 previous similar message [ 8286.239247] LNet: 5 local NIs in recovery (showing 5): 192.168.201.105@tcp41, 192.168.201.105@tcp46, 192.168.201.105@tcp44, 192.168.201.105@tcp43, 192.168.201.105@tcp42 [ 8290.291983] LustreError: 504521:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 8290.332440] Lustre: server umount lustre-OST0000 complete [ 8290.337087] LNetError: 479599:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.105 [ 8290.340781] LNetError: 479599:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 39 previous similar messages [ 8290.345593] LNetError: Protocol error connecting to 192.168.201.105@tcp42 at host 192.168.201.105:988: is it running a compatible version of Lustre? [ 8290.348583] LNetError: Skipped 39 previous similar messages [ 8295.905885] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8295.905917] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8295.906759] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8295.912317] Lustre: Skipped 1 previous similar message [ 8295.914717] LustreError: Skipped 1 previous similar message [ 8297.727824] LustreError: 504726:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 8297.730558] LustreError: 504726:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8297.761968] Lustre: server umount lustre-MDT0000 complete [ 8298.968422] LustreError: 480332:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764741497 with bad export cookie 4922393969652557957 [ 8298.972601] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8298.972972] LustreError: 480332:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8299.033662] LustreError: 504928:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8299.036657] LustreError: 504928:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8299.092427] Lustre: server umount lustre-MDT0001 complete [ 8302.541537] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 8303.538906] Key type lgssc unregistered [ 8303.695034] LNet: 505466:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8303.697729] LNetError: 505466:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8303.699518] LNet: Removed LNI 192.168.201.105@tcp [ 8304.735223] LNet: Removed LNI 192.168.201.105@tcp10 [ 8304.737438] LNetError: 505466:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8304.740963] LNetError: 505466:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 8304.742954] LNet: Removed LNI 192.168.201.105@tcp11 [ 8305.759175] LNet: Removed LNI 192.168.201.105@tcp12 [ 8305.760289] LNetError: 505466:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8305.762266] LNetError: 505466:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 8305.764364] LNet: Removed LNI 192.168.201.105@tcp13 [ 8306.783168] LNet: Removed LNI 192.168.201.105@tcp14 [ 8307.807142] LNet: Removed LNI 192.168.201.105@tcp15 [ 8307.808407] LNetError: 505466:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8307.810052] LNetError: 505466:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 8307.812164] LNet: Removed LNI 192.168.201.105@tcp16 [ 8308.831137] LNet: Removed LNI 192.168.201.105@tcp17 [ 8309.855093] LNet: Removed LNI 192.168.201.105@tcp18 [ 8310.879171] LNet: Removed LNI 192.168.201.105@tcp19 [ 8311.903141] LNet: Removed LNI 192.168.201.105@tcp20 [ 8311.904630] LNetError: 505466:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8311.906341] LNetError: 505466:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 8311.909686] LNet: Removed LNI 192.168.201.105@tcp21 [ 8312.927163] LNet: Removed LNI 192.168.201.105@tcp22 [ 8313.951199] LNet: Removed LNI 192.168.201.105@tcp23 [ 8314.975188] LNet: Removed LNI 192.168.201.105@tcp24 [ 8315.999204] LNet: Removed LNI 192.168.201.105@tcp25 [ 8317.023217] LNet: Removed LNI 192.168.201.105@tcp26 [ 8318.047219] LNet: Removed LNI 192.168.201.105@tcp27 [ 8319.071643] LNet: Removed LNI 192.168.201.105@tcp28 [ 8320.095220] LNet: Removed LNI 192.168.201.105@tcp29 [ 8320.097114] LNetError: 505466:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8320.100223] LNetError: 505466:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [ 8320.103662] LNet: Removed LNI 192.168.201.105@tcp30 [ 8321.119132] LNet: Removed LNI 192.168.201.105@tcp31 [ 8322.143157] LNet: Removed LNI 192.168.201.105@tcp32 [ 8323.167154] LNet: Removed LNI 192.168.201.105@tcp33 [ 8324.191232] LNet: Removed LNI 192.168.201.105@tcp34 [ 8325.215212] LNet: Removed LNI 192.168.201.105@tcp35 [ 8326.239236] LNet: Removed LNI 192.168.201.105@tcp36 [ 8327.263147] LNet: Removed LNI 192.168.201.105@tcp37 [ 8328.287126] LNet: Removed LNI 192.168.201.105@tcp38 [ 8329.311103] LNet: Removed LNI 192.168.201.105@tcp39 [ 8330.335154] LNet: Removed LNI 192.168.201.105@tcp40 [ 8330.336565] LNet: Removed LNI 192.168.201.105@tcp41 [ 8330.337933] LNet: Removed LNI 192.168.201.105@tcp42 [ 8330.339664] LNet: Removed LNI 192.168.201.105@tcp43 [ 8330.341027] LNet: Removed LNI 192.168.201.105@tcp44 [ 8331.359132] LNet: Removed LNI 192.168.201.105@tcp45 [ 8331.360425] LNet: Removed LNI 192.168.201.105@tcp46 [ 8332.383131] LNet: Removed LNI 192.168.201.105@tcp47 [ 8333.407142] LNet: Removed LNI 192.168.201.105@tcp48 [ 8334.431216] LNet: Removed LNI 192.168.201.105@tcp49 [ 8334.438370] LNet: Removed LNI 192.168.201.105@tcp50 [ 8334.691992] LNetError: 505530:0:(module.c:692:libcfs_exit()) Portals memory leaked: 376 bytes [ 8334.710700] Key type .llcrypt unregistered [ 8334.711775] Key type ._llcrypt unregistered [ 8341.888559] Key type ._llcrypt registered [ 8341.889550] Key type .llcrypt registered [ 8341.921758] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 8342.383645] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8342.394502] alg: No test for adler32 (adler32-zlib) [ 8343.234699] Lustre: Lustre: Build Version: 2.17.0_RC1_3_g509e2bd [ 8343.316974] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 8344.903264] Key type lgssc registered [ 8345.291682] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8347.749890] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8347.756386] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8348.906395] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8350.211769] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 8352.843306] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8352.948343] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8354.143932] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 8355.778447] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8356.878282] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8358.940460] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8359.048259] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8360.744560] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 8362.368655] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8366.779731] loop0: detected capacity change from 0 to 204800000 [ 8366.853118] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8366.856377] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8366.859160] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8366.881844] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8368.775569] loop0: detected capacity change from 0 to 204800000 [ 8368.832808] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8368.837674] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8368.842597] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8368.898065] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8370.534954] loop0: detected capacity change from 0 to 204800000 [ 8372.447482] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8372.470727] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8372.515401] Lustre: fs15246-OST0000: new disk, initializing [ 8372.518638] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [ 8372.540307] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8374.478827] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 8378.582605] loop1: detected capacity change from 0 to 204800000 [ 8380.538140] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8380.562843] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8380.598373] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [ 8380.622829] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [ 8380.661996] Lustre: fs15246-MDT0000: new disk, initializing [ 8380.720489] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8380.733662] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8380.781147] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 8381.793174] Lustre: 511067:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@00000000b30c6f8b x1850465626124928/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1764741596 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 8382.173414] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 8390.879259] LustreError: 511067:0:(fid_request.c:213:seq_client_alloc_seq()) cli-cli-fs15246-OST0000-osc-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 8390.882080] LustreError: 511067:0:(fid_request.c:255:seq_client_get_seq()) cli-cli-fs15246-OST0000-osc-MDT0000: Can't allocate new sequence: rc = -5 [ 8390.884729] LustreError: 511067:0:(osp_precreate.c:459:osp_precreate_rollover_new_seq()) fs15246-OST0000-osc-MDT0000: alloc fid error: rc = -5 [ 8394.216629] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8394.230640] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [ 8396.091224] LustreError: 512192:0:(obd_class.h:479:obd_check_dev()) Device 30 not setup [ 8396.117058] Lustre: server umount fs15246-MDT0000 complete [ 8399.322116] LustreError: 512694:0:(obd_class.h:479:obd_check_dev()) Device 32 not setup [ 8399.325581] LustreError: 512694:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8399.343475] Lustre: server umount fs15246-OST0000 complete [ 8404.450911] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8404.451119] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8404.453756] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8404.456673] Lustre: Skipped 1 previous similar message [ 8404.460093] Lustre: Skipped 1 previous similar message [ 8409.571219] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8409.573035] Lustre: Skipped 1 previous similar message [ 8414.689485] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8414.691100] Lustre: Skipped 1 previous similar message [ 8417.247077] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8417.264630] LustreError: 513194:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8417.266625] LustreError: 513194:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 8417.298783] Lustre: server umount lustre-OST0000 complete [ 8419.809769] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8419.810414] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8419.812963] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8419.816697] Lustre: Skipped 1 previous similar message [ 8424.634983] LustreError: 513397:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 8424.637447] LustreError: 513397:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8424.666021] Lustre: server umount lustre-MDT0000 complete [ 8424.928483] LustreError: 506691:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8424.935333] LustreError: 506691:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 8425.824121] LustreError: 506671:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764741624 with bad export cookie 16337648820757431294 [ 8425.826954] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8425.829455] LustreError: 506671:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8425.943392] Lustre: server umount lustre-MDT0001 complete [ 8429.295637] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 8430.192419] Key type lgssc unregistered [ 8430.319385] LNet: 514131:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8430.321935] LNetError: 514131:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8430.330327] LNet: Removed LNI 192.168.201.105@tcp [ 8430.625882] Key type .llcrypt unregistered [ 8430.627457] Key type ._llcrypt unregistered [ 8437.958044] Key type ._llcrypt registered [ 8437.959269] Key type .llcrypt registered [ 8437.997567] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 8438.346665] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8438.389240] alg: No test for adler32 (adler32-zlib) [ 8439.252574] Lustre: Lustre: Build Version: 2.17.0_RC1_3_g509e2bd [ 8439.336837] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 8440.919127] Key type lgssc registered [ 8441.287029] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8443.902077] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8445.621033] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8447.480117] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8449.069480] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8453.945792] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [ 8457.041891] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 8460.010624] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8462.143414] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8464.367310] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8466.545043] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8468.887488] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8468.907204] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8468.912804] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8469.999132] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8470.012112] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8470.057302] Lustre: lustre-MDT0000: new disk, initializing [ 8470.081870] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8470.088717] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8471.478436] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 8475.856660] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8475.886441] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8475.921357] Lustre: 518546:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8475.947630] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8475.951141] Lustre: Skipped 1 previous similar message [ 8475.993053] Lustre: lustre-MDT0001: new disk, initializing [ 8476.032687] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8476.047502] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8476.053138] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8477.312441] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 8480.335944] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8481.541596] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8483.799981] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8483.822548] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8483.915837] Lustre: lustre-OST0000: new disk, initializing [ 8483.918358] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8483.949283] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8485.703854] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 8488.725131] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8489.462911] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8489.466098] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8489.497146] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8489.966988] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8490.025733] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8491.231763] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8491.296147] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8494.561399] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8494.561768] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8494.565662] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8494.567835] LustreError: Skipped 1 previous similar message [ 8497.964132] LustreError: 521481:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 8498.009385] Lustre: server umount lustre-OST0000 complete [ 8499.681397] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8499.682058] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8499.685456] Lustre: Skipped 2 previous similar messages [ 8499.689133] Lustre: Skipped 1 previous similar message [ 8504.800819] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8504.803260] Lustre: Skipped 1 previous similar message [ 8505.474663] LustreError: 521682:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 8505.477842] LustreError: 521682:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8505.511967] Lustre: server umount lustre-MDT0000 complete [ 8506.812401] LustreError: 519327:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764741705 with bad export cookie 8825815394075596677 [ 8506.816108] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8506.816773] LustreError: 519327:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8506.888432] LustreError: 521883:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8506.891833] LustreError: 521883:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8506.948160] Lustre: server umount lustre-MDT0001 complete [ 8515.250062] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [ 8518.693866] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 8522.386231] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8524.847183] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8527.228476] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8529.683574] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8530.306661] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 8243 sec ========== 01:02:08 (1764741728) [ 8530.945457] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 01:02:09 (1764741729) === [ 8532.170621] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 01:02:10 (1764741730) === [ 8546.361045] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 8547.493773] Key type lgssc unregistered [ 8547.633427] LNet: 526731:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8547.636140] LNetError: 526731:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8547.644305] LNet: Removed LNI 192.168.201.105@tcp [ 8547.978243] Key type .llcrypt unregistered [ 8547.979704] Key type ._llcrypt unregistered