[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 512021366 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2400.000 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2439 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22D5 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 002295 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2349 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23D9 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2411 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22d5-0xbffe2348] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22d4] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2349-0xbffe23d8] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23d9-0xbffe2410] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2411-0xbffe2438] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003292] x2apic enabled [ 0.004009] Switched APIC routing to physical x2apic. [ 0.005017] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 0.008021] Calibrating delay loop (skipped) preset value.. 4800.00 BogoMIPS (lpj=2400000) [ 0.009013] pid_max: default: 32768 minimum: 301 [ 0.011046] LSM: Security Framework initializing [ 0.012056] Yama: becoming mindful. [ 0.013046] SELinux: Initializing. [ 0.014067] *** VALIDATE selinux *** [ 0.022491] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027529] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028148] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030010] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031125] *** VALIDATE tmpfs *** [ 0.033333] *** VALIDATE proc *** [ 0.034240] *** VALIDATE cgroup *** [ 0.035013] *** VALIDATE cgroup2 *** [ 0.036237] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037160] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039034] Spectre V2 : User space: Vulnerable [ 0.040011] Speculative Store Bypass: Vulnerable [ 0.042754] debug: unmapping init [mem 0xffffffffbd459000-0xffffffffbd460fff] [ 0.044243] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045659] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046024] ... version: 2 [ 0.047012] ... bit width: 48 [ 0.048014] ... generic registers: 4 [ 0.049012] ... value mask: 0000ffffffffffff [ 0.050013] ... max period: 00007fffffffffff [ 0.051015] ... fixed-purpose events: 3 [ 0.052010] ... event mask: 000000070000000f [ 0.053317] rcu: Hierarchical SRCU implementation. [ 0.055433] smp: Bringing up secondary CPUs ... [ 0.056584] x86: Booting SMP configuration: [ 0.057025] .... node #0, CPUs: #1 #2 #3 [ 0.061044] smp: Brought up 1 node, 4 CPUs [ 0.063030] smpboot: Max logical packages: 1 [ 0.064020] smpboot: Total of 4 processors activated (19200.00 BogoMIPS) [ 0.202343] node 0 deferred pages initialised in 136ms [ 0.206029] devtmpfs: initialized [ 0.208347] x86/mm: Memory block size: 128MB [ 0.212025] gcov: version magic: 0x41383552 [ 0.217499] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.218065] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.219368] pinctrl core: initialized pinctrl subsystem [ 0.221316] [ 0.221931] ************************************************************* [ 0.225020] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.227013] ** ** [ 0.230016] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.232014] ** ** [ 0.235020] ** This means that this kernel is built to expose internal ** [ 0.237015] ** IOMMU data structures, which may compromise security on ** [ 0.239013] ** your system. ** [ 0.241014] ** ** [ 0.243013] ** If you see this message and you are not debugging the ** [ 0.245014] ** kernel, report this immediately to your vendor! ** [ 0.247018] ** ** [ 0.249034] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.251014] ************************************************************* [ 0.254649] NET: Registered protocol family 16 [ 0.256518] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.258065] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.261090] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.265139] cpuidle: using governor menu [ 0.267294] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.269572] PCI: Using configuration type 1 for base access [ 0.272220] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.281092] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.282058] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.284078] cryptd: max_cpu_qlen set to 1000 [ 0.286312] ACPI: Added _OSI(Module Device) [ 0.287024] ACPI: Added _OSI(Processor Device) [ 0.288237] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.289043] ACPI: Added _OSI(Processor Aggregator Device) [ 0.294535] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.302368] ACPI: Interpreter enabled [ 0.304086] ACPI: PM: (supports S0 S3 S4 S5) [ 0.306016] ACPI: Using IOAPIC for interrupt routing [ 0.308138] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.311612] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.322084] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.324064] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.327027] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.330107] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.335593] acpiphp: Slot [2] registered [ 0.337174] acpiphp: Slot [5] registered [ 0.339231] acpiphp: Slot [6] registered [ 0.340148] acpiphp: Slot [7] registered [ 0.342157] acpiphp: Slot [8] registered [ 0.344145] acpiphp: Slot [9] registered [ 0.345134] acpiphp: Slot [10] registered [ 0.347167] acpiphp: Slot [3] registered [ 0.350146] acpiphp: Slot [4] registered [ 0.351132] acpiphp: Slot [11] registered [ 0.353150] acpiphp: Slot [12] registered [ 0.354280] acpiphp: Slot [13] registered [ 0.356092] acpiphp: Slot [14] registered [ 0.357215] acpiphp: Slot [15] registered [ 0.358081] acpiphp: Slot [16] registered [ 0.360138] acpiphp: Slot [17] registered [ 0.361126] acpiphp: Slot [18] registered [ 0.362117] acpiphp: Slot [19] registered [ 0.364152] acpiphp: Slot [20] registered [ 0.366276] acpiphp: Slot [21] registered [ 0.368172] acpiphp: Slot [22] registered [ 0.370120] acpiphp: Slot [23] registered [ 0.371103] acpiphp: Slot [24] registered [ 0.372124] acpiphp: Slot [25] registered [ 0.373116] acpiphp: Slot [26] registered [ 0.376150] acpiphp: Slot [27] registered [ 0.377096] acpiphp: Slot [28] registered [ 0.378210] acpiphp: Slot [29] registered [ 0.380087] acpiphp: Slot [30] registered [ 0.381112] acpiphp: Slot [31] registered [ 0.382067] PCI host bridge to bus 0000:00 [ 0.384022] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.386027] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.388031] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.390029] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.392030] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.395031] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.397219] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.400093] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.402499] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.412020] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.417548] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.421026] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.423024] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.426027] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.429656] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.433146] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.436058] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.440074] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.444023] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.458024] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.463988] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.471364] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.479019] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.488022] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.504051] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.512880] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.520020] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.527022] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.546021] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.555634] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.562020] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.567018] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.584022] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.593246] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.599018] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.613023] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.648020] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.657218] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.663016] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.671019] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.684032] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.694794] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.700020] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.706017] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.720018] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.729881] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.733440] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.735453] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.738454] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.741293] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.746039] iommu: Default domain type: Passthrough [ 0.748435] SCSI subsystem initialized [ 0.750143] ACPI: bus type USB registered [ 0.752131] usbcore: registered new interface driver usbfs [ 0.754266] usbcore: registered new interface driver hub [ 0.756087] usbcore: registered new device driver usb [ 0.759222] pps_core: LinuxPPS API ver. 1 registered [ 0.761023] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.765067] PTP clock support registered [ 0.767125] EDAC MC: Ver: 3.0.0 [ 0.769140] PCI: Using ACPI for IRQ routing [ 0.772133] NetLabel: Initializing [ 0.774016] NetLabel: domain hash size = 128 [ 0.776028] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.779116] NetLabel: unlabeled traffic allowed by default [ 0.782057] vgaarb: loaded [ 0.783295] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.787021] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.796121] clocksource: Switched to clocksource kvm-clock [ 0.909831] VFS: Disk quotas dquot_6.6.0 [ 0.911240] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.913316] *** VALIDATE ramfs *** [ 0.914256] *** VALIDATE hugetlbfs *** [ 0.915424] pnp: PnP ACPI init [ 0.917482] pnp: PnP ACPI: found 6 devices [ 0.944177] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.946800] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.948273] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.950440] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.953310] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.956210] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.959318] NET: Registered protocol family 2 [ 0.962579] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.968505] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.972455] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.978545] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.982290] TCP: Hash tables configured (established 65536 bind 65536) [ 0.987455] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.990875] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.993866] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.997226] NET: Registered protocol family 1 [ 1.000561] RPC: Registered named UNIX socket transport module. [ 1.003481] RPC: Registered udp transport module. [ 1.005178] RPC: Registered tcp transport module. [ 1.006847] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.009505] NET: Registered protocol family 44 [ 1.011228] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.013664] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.015883] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.018743] PCI: CLS 0 bytes, default 64 [ 1.020873] Unpacking initramfs... [ 2.454853] debug: unmapping init [mem 0xffff89b0bcc54000-0xffff89b0bffbffff] [ 2.460043] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.462596] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.465932] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 2.952533] Initialise system trusted keyrings [ 2.954531] Key type blacklist registered [ 2.956708] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.966347] zbud: loaded [ 2.969545] *** VALIDATE nfs *** [ 2.970901] *** VALIDATE nfs4 *** [ 2.972348] pstore: using deflate compression [ 2.977091] Platform Keyring initialized [ 3.084935] NET: Registered protocol family 38 [ 3.086795] Key type asymmetric registered [ 3.088532] Asymmetric key parser 'x509' registered [ 3.090533] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.094122] io scheduler mq-deadline registered [ 3.095889] io scheduler kyber registered [ 3.097737] io scheduler bfq registered [ 3.100299] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.103566] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.107180] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.110285] ACPI: Power Button [PWRF] [ 3.115522] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.122565] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.139062] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.151161] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.184219] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.215145] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.245391] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.251361] Non-volatile memory driver v1.3 [ 3.253412] Linux agpgart interface v0.103 [ 3.287437] virtio_blk virtio1: [vda] 134608 512-byte logical blocks (68.9 MB/65.7 MiB) [ 3.290788] vda: detected capacity change from 0 to 68919296 [ 3.305445] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.308916] vdb: detected capacity change from 0 to 1073741824 [ 3.326663] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.330056] vdc: detected capacity change from 0 to 2621440000 [ 3.345552] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.349244] vdd: detected capacity change from 0 to 2621440000 [ 3.372407] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.379224] vde: detected capacity change from 0 to 4294967296 [ 3.397680] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.401064] vdf: detected capacity change from 0 to 4294967296 [ 3.410421] libphy: Fixed MDIO Bus: probed [ 3.439535] usbcore: registered new interface driver usbserial_generic [ 3.442446] usbserial: USB Serial support registered for generic [ 3.446482] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.452413] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.454780] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.457964] mousedev: PS/2 mouse device common for all mice [ 3.465580] rtc_cmos 00:05: RTC can wake from S4 [ 3.465589] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.471149] rtc_cmos 00:05: registered as rtc0 [ 3.478277] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.478319] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.483498] intel_pstate: CPU model not supported [ 3.487923] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.492665] hid: raw HID events driver (C) Jiri Kosina [ 3.495481] usbcore: registered new interface driver usbhid [ 3.497399] usbhid: USB HID core driver [ 3.499404] drop_monitor: Initializing network drop monitor service [ 3.502022] Initializing XFRM netlink socket [ 3.504156] NET: Registered protocol family 10 [ 3.507651] Segment Routing with IPv6 [ 3.509097] NET: Registered protocol family 17 [ 3.512490] mpls_gso: MPLS GSO support [ 3.519138] RAS: Correctable Errors collector initialized. [ 3.521693] AVX version of gcm_enc/dec engaged. [ 3.524276] AES CTR mode by8 optimization enabled [ 3.609418] sched_clock: Marking stable (3609384861, 0)->(4540707148, -931322287) [ 3.613678] registered taskstats version 1 [ 3.616298] Loading compiled-in X.509 certificates [ 3.618786] zswap: loaded using pool lzo/zbud [ 3.648118] Key type big_key registered [ 3.662936] Key type encrypted registered [ 3.664937] ima: No TPM chip found, activating TPM-bypass! [ 3.667635] ima: Allocated hash algorithm: sha1 [ 3.670519] ima: No architecture policies found [ 3.673182] evm: Initialising EVM extended attributes: [ 3.675542] evm: security.selinux [ 3.677547] evm: security.ima [ 3.678752] evm: security.capability [ 3.680529] evm: HMAC attrs: 0x1 [ 3.683418] rtc_cmos 00:05: setting system clock to 2026-04-29 21:05:42 UTC (1777496742) [ 3.690965] debug: unmapping init [mem 0xffffffffbe403000-0xffffffffbe5fffff] [ 3.694448] debug: unmapping init [mem 0xffffffffbd182000-0xffffffffbd458fff] [ 3.702092] Write protecting the kernel read-only data: 28672k [ 3.705599] debug: unmapping init [mem 0xffffffffbb803000-0xffffffffbb9fffff] [ 3.709756] debug: unmapping init [mem 0xffffffffbc114000-0xffffffffbc1fffff] [ 3.785566] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.795403] systemd[1]: Detected virtualization kvm. [ 3.797418] systemd[1]: Detected architecture x86-64. [ 3.800028] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.825739] systemd[1]: No hostname configured. [ 3.827456] systemd[1]: Set hostname to . [ 3.829342] random: systemd: uninitialized urandom read (16 bytes read) [ 3.831566] systemd[1]: Initializing machine ID from random generator. [ 3.898736] random: ln: uninitialized urandom read (6 bytes read) [ 4.001228] random: systemd: uninitialized urandom read (16 bytes read) [ 4.004734] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 4.011191] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 4.021588] systemd[1]: Starting Apply Kernel Variables... Starting Apply Kernel Variables... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Swap. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Paths. [ OK ] Reached target Timers. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Initrd Root Device. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Sockets. Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.664446] device-mapper: uevent: version 1.0.3 [ 4.669062] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.404939] virtio_net virtio0 ens2: renamed from eth0 [ 5.445130] scsi host0: ata_piix [ 5.462762] scsi host1: ata_piix [ 5.464127] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.466274] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 6.112960] random: fast init done [ 9.766566] dracut-initqueue[592]: RTNETLINK answers: File exists [ 10.190518] random: crng init done [ 10.192202] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.686406] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.936455] printk: systemd: 25 output lines suppressed due to ratelimiting [ 12.211708] SELinux: Disabled at runtime. [ 12.274936] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.284182] systemd[1]: Detected virtualization kvm. [ 12.286155] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.768060] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.771361] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.775598] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.780324] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.783888] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.793434] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.797621] systemd[1]: Started Forward Password Requests to Wall Directory Watch. [ OK ] Started Forward Password Requests to Wall Directory Watch. Mounting Kernel Debug File System... Mounting POSIX Message Queue File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Listening on udev Control Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Reached target Local Encrypted Volumes. Starting udev Coldplug all Devices... [ OK ] Created slice system-getty.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Apply Kernel Variables... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on Process Core Dump Socket. Activating swap /dev/disk/by-label/SWAP... [ OK ] Stopped target Initrd File Systems. Mounting Huge Pages File System... [ OK ] Listening on initctl Compatibility Named Pipe. [ 12.924824] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Remount Root and Kernel File Systems... [ OK ] Reached target rpc_pipefs.target. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.237838] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.540336] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.594863] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.721195] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.735655] EDAC sbridge: Ver: 1.1.2 [ 15.506949] Key type dns_resolver registered [ 15.804401] NFS: Registering the id_resolver key type [ 15.807030] Key type id_resolver registered [ 15.808981] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. [ OK ] Reached target sshd-keygen.target. Starting Network Manager... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg406-server login: [ 42.090811] libcfs: loading out-of-tree module taints kernel. [ 42.109154] Key type ._llcrypt registered [ 42.110990] Key type .llcrypt registered [ 42.163171] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_hostid [ 59.554007] hrtimer: interrupt took 2180648 ns [ 60.225320] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 61.945986] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 61.957992] alg: No test for adler32 (adler32-zlib) [ 63.376815] Lustre: Lustre: Build Version: 2.17.52_93_g6cdb58a [ 64.678800] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 66.512372] Key type lgssc registered [ 68.964676] Lustre: Echo OBD driver; http://www.lustre.org/ [ 85.963063] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 130.566160] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 145.641451] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 145.677548] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 147.156703] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 147.200846] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 147.341668] Lustre: lustre-MDT0000: new disk, initializing [ 147.566294] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 147.621807] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 153.085957] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 169.764031] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 169.891885] Lustre: 6507:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 169.940355] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 169.949364] Lustre: Skipped 1 previous similar message [ 170.063803] Lustre: lustre-MDT0001: new disk, initializing [ 170.144041] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 170.220321] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 170.240223] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 175.278499] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 180.475734] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 191.627361] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 192.034780] Lustre: lustre-OST0000: new disk, initializing [ 192.042725] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 192.142143] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 194.586245] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 194.591267] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 194.696535] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 199.407144] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 216.263636] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 216.349918] Lustre: lustre-OST0001: new disk, initializing [ 216.352890] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 216.474126] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 223.874298] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 226.315087] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 226.324592] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 231.430480] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 236.606818] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 244.000657] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 250.327543] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing check_logdir /tmp/testlogs/ [ 254.484861] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing yml_node [ 259.414351] Lustre: DEBUG MARKER: Client: 2.17.52.93 [ 261.724302] Lustre: DEBUG MARKER: MDS: 2.17.52.93 [ 264.486123] Lustre: DEBUG MARKER: OSS: 2.17.52.93 [ 266.180948] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Wed Apr 29 17:10:03 EDT 2026 [ 283.783903] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 286.226074] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 303.083373] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 303.093477] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 303.099355] Lustre: Skipped 1 previous similar message [ 303.123338] Lustre: Skipped 3 previous similar messages [ 308.202257] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 308.207978] Lustre: Skipped 3 previous similar messages [ 308.681936] Lustre: server umount lustre-MDT0000 complete [ 312.498977] LustreError: 6497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777497051 with bad export cookie 3250119924490344656 [ 312.502696] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 312.516631] LustreError: 6497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 313.313472] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 313.329743] Lustre: Skipped 2 previous similar messages [ 313.334952] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 313.339542] Lustre: Skipped 1 previous similar message [ 317.345519] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 317.349768] Lustre: Skipped 1 previous similar message [ 318.996400] Lustre: server umount lustre-MDT0001 complete [ 323.106047] Lustre: server umount lustre-OST0000 complete [ 327.477523] Lustre: server umount lustre-OST0001 complete [ 334.429077] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_hostid [ 344.213826] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 384.002370] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 384.310746] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 384.338094] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 384.427267] Lustre: lustre-MDT0000: new disk, initializing [ 384.539829] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 384.580877] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 388.822515] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 399.372608] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 399.487779] Lustre: 14982:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 399.569292] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 399.574743] Lustre: Skipped 1 previous similar message [ 399.682537] Lustre: lustre-MDT0001: new disk, initializing [ 399.775782] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 399.814621] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 399.823442] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 403.543405] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 410.627711] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 413.884181] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 419.903600] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 420.302121] Lustre: lustre-OST0000: new disk, initializing [ 420.312473] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 420.432344] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 421.633840] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 421.642691] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 421.845463] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 426.289479] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 433.884724] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 438.881952] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 439.233328] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 443.762480] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 443.961663] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 447.465464] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 447.466284] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 447.468118] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 447.494509] Lustre: Skipped 2 previous similar messages [ 451.939578] Lustre: server umount lustre-OST0000 complete [ 456.177155] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 456.196922] LustreError: Skipped 1 previous similar message [ 456.204374] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 456.229703] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 456.235430] Lustre: Skipped 1 previous similar message [ 461.271157] Lustre: server umount lustre-MDT0000 complete [ 462.820798] LustreError: 15876:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 462.848242] LustreError: 15876:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 5 previous similar messages [ 464.639566] LustreError: 14973:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777497203 with bad export cookie 3250119924490346189 [ 464.649247] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 464.650906] LustreError: 14973:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 465.108489] Lustre: server umount lustre-MDT0001 complete [ 468.839636] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 17:13:25 (1777497205) [ 471.085150] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 472.631596] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 17:13:29 (1777497209) [ 478.778249] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 479.268703] LustreError: 18988:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 479.396758] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 483.658779] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 484.834515] LustreError: 18989:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 489.958019] LustreError: 18988:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 490.835253] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 494.613680] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 499.891728] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 503.804746] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 510.245332] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 516.566959] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 522.374164] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 535.528988] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 535.533486] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 535.545588] Lustre: Skipped 1 previous similar message [ 535.548112] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 535.572822] Lustre: Skipped 2 previous similar messages [ 540.524065] Lustre: server umount lustre-OST0000 complete [ 545.762725] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 545.787821] Lustre: Skipped 2 previous similar messages [ 550.030300] Lustre: server umount lustre-MDT0000 complete [ 550.884797] LustreError: 18985:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 550.907385] LustreError: 18985:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 553.426826] LustreError: 19704:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777497292 with bad export cookie 3250119924490347071 [ 553.431810] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 553.438720] LustreError: 19704:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 553.789203] Lustre: server umount lustre-MDT0001 complete [ 564.606481] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 567.198800] Key type lgssc unregistered [ 567.537806] LNet: 22177:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 567.553336] LNetError: 22177:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 567.582939] LNet: Removed LNI 192.168.204.106@tcp [ 568.592167] Key type .llcrypt unregistered [ 568.596928] Key type ._llcrypt unregistered [ 591.985480] Key type ._llcrypt registered [ 591.988079] Key type .llcrypt registered [ 592.131499] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 593.179823] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 593.397526] alg: No test for adler32 (adler32-zlib) [ 594.529700] Lustre: Lustre: Build Version: 2.17.52_93_g6cdb58a [ 594.722806] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 596.360506] Key type lgssc registered [ 597.874956] Lustre: Echo OBD driver; http://www.lustre.org/ [ 607.430708] loop: module loaded [ 607.522372] loop0: detected capacity change from 0 to 204800000 [ 607.829986] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 607.854152] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 608.316077] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 614.493354] loop0: detected capacity change from 0 to 204800000 [ 614.762694] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 614.773660] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 614.782204] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 620.993074] loop0: detected capacity change from 0 to 204800000 [ 627.031734] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 627.057991] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 628.382965] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 628.496026] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 628.580270] Lustre: lustre-MDT0000: new disk, initializing [ 628.690267] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 628.713697] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 632.952618] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 641.808771] loop1: detected capacity change from 0 to 204800000 [ 647.690275] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 648.090540] Lustre: lustre-OST07c6: new disk, initializing [ 648.093204] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 648.096811] Lustre: Skipped 1 previous similar message [ 648.211603] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 654.552421] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 656.513932] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 656.523098] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 656.585415] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 660.018133] Lustre: 25828:0:(osd_internal.h:1447:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 660.029373] Lustre: 25828:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 660.037908] Lustre: 25828:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 660.047165] Lustre: 25828:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 660.060510] Lustre: 25828:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 660.072437] Lustre: 25828:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 674.612347] Lustre: 24330:0:(osd_internal.h:1447:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 674.630335] Lustre: 24330:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 674.642969] Lustre: 24330:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 674.648346] Lustre: 24330:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 674.652288] Lustre: 24330:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 674.657294] Lustre: 24330:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 679.570815] Lustre: 25828:0:(osd_internal.h:1447:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 679.579775] Lustre: 25828:0:(osd_internal.h:1447:osd_trans_exec_op()) Skipped 1 previous similar message [ 679.587188] Lustre: 25828:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 679.599690] Lustre: 25828:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 679.611525] Lustre: 25828:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 679.633974] Lustre: 25828:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 679.646232] Lustre: 25828:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 679.658797] Lustre: 25828:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 679.665366] Lustre: 25828:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 679.672179] Lustre: 25828:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 679.679681] Lustre: 25828:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 679.691929] Lustre: 25828:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 682.915877] Lustre: 24333:0:(osd_internal.h:1447:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 682.925764] Lustre: 24333:0:(osd_internal.h:1447:osd_trans_exec_op()) Skipped 1 previous similar message [ 682.931782] Lustre: 24333:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 682.941552] Lustre: 24333:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 682.945850] Lustre: 24333:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 682.954374] Lustre: 24333:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 682.970044] Lustre: 24333:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/0 [ 682.977368] Lustre: 24333:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 682.988038] Lustre: 24333:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 682.994019] Lustre: 24333:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 683.000257] Lustre: 24333:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 683.012082] Lustre: 24333:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 687.955233] Lustre: 24333:0:(osd_internal.h:1447:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 687.975376] Lustre: 24333:0:(osd_internal.h:1447:osd_trans_exec_op()) Skipped 1 previous similar message [ 687.981611] Lustre: 24333:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 687.990766] Lustre: 24333:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 687.996928] Lustre: 24333:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 688.007061] Lustre: 24333:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 688.022528] Lustre: 24333:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 688.030572] Lustre: 24333:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 688.037305] Lustre: 24333:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 688.041531] Lustre: 24333:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 688.045151] Lustre: 24333:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 688.049740] Lustre: 24333:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 699.443184] Lustre: 24332:0:(osd_internal.h:1447:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 699.463419] Lustre: 24332:0:(osd_internal.h:1447:osd_trans_exec_op()) Skipped 5 previous similar messages [ 699.475356] Lustre: 24332:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 699.490574] Lustre: 24332:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 699.502765] Lustre: 24332:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 699.508971] Lustre: 24332:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 699.516766] Lustre: 24332:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 699.523752] Lustre: 24332:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 699.531760] Lustre: 24332:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 699.536737] Lustre: 24332:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 699.546854] Lustre: 24332:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 699.555618] Lustre: 24332:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 748.515159] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 748.541881] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 753.639878] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 754.483567] Lustre: server umount lustre-OST07c6 complete [ 766.966096] Lustre: server umount lustre-MDT0000 complete [ 784.526814] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 787.432172] Key type lgssc unregistered [ 787.854842] LNet: 27852:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 787.870824] LNetError: 27852:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 788.903510] LNet: Removed LNI 192.168.204.106@tcp [ 789.875548] Key type .llcrypt unregistered [ 789.878070] Key type ._llcrypt unregistered [ 808.501441] Key type ._llcrypt registered [ 808.505247] Key type .llcrypt registered [ 808.622932] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 809.940302] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 809.964509] alg: No test for adler32 (adler32-zlib) [ 811.044510] Lustre: Lustre: Build Version: 2.17.52_93_g6cdb58a [ 811.210297] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 812.872210] Key type lgssc registered [ 813.965761] Lustre: Echo OBD driver; http://www.lustre.org/ [ 822.468360] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 822.508261] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 824.095858] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 828.206501] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 833.506632] LustreError: 29106:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 833.519759] LustreError: 29106:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 837.587063] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 838.081504] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 843.110473] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 848.911377] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 853.034526] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 859.630725] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 859.998627] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 865.906359] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 868.224074] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 871.467267] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 888.801575] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 888.804749] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 888.826951] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 889.294833] Lustre: server umount lustre-OST0000 complete [ 893.925246] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 893.939820] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 893.949032] Lustre: Skipped 1 previous similar message [ 893.949453] Lustre: Skipped 2 previous similar messages [ 899.045747] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 899.051244] Lustre: Skipped 2 previous similar messages [ 899.823831] Lustre: server umount lustre-MDT0000 complete [ 904.163729] LustreError: 29087:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777497642 with bad export cookie 12859127937289505991 [ 904.167297] LustreError: 29106:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 904.180841] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 904.189041] LustreError: 29087:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 904.652279] Lustre: server umount lustre-MDT0001 complete [ 916.165776] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 919.098255] Key type lgssc unregistered [ 919.432736] LNet: 32486:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 919.449702] LNetError: 32486:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 919.476181] LNet: Removed LNI 192.168.204.106@tcp [ 920.218084] Key type .llcrypt unregistered [ 920.220994] Key type ._llcrypt unregistered [ 939.323621] Key type ._llcrypt registered [ 939.325828] Key type .llcrypt registered [ 939.441119] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 940.368806] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 940.400202] alg: No test for adler32 (adler32-zlib) [ 941.525118] Lustre: Lustre: Build Version: 2.17.52_93_g6cdb58a [ 941.838156] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 943.512264] Key type lgssc registered [ 944.676735] Lustre: Echo OBD driver; http://www.lustre.org/ [ 952.410978] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 952.446534] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 954.239422] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 958.778374] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 963.561119] LustreError: 33742:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 963.578340] LustreError: 33742:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 966.818242] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 967.105423] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 970.932168] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 975.941060] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 979.885828] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 987.019993] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 987.572498] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 993.982551] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 998.839354] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 999.454213] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1012.195102] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1012.203094] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1012.240896] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1014.248088] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1014.250394] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1014.265153] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1014.291619] Lustre: Skipped 1 previous similar message [ 1016.892974] Lustre: server umount lustre-OST0000 complete [ 1023.459079] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1023.470612] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1023.492914] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1027.396910] Lustre: server umount lustre-MDT0000 complete [ 1029.601424] LustreError: 33743:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1029.631133] LustreError: 33743:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 1031.210272] LustreError: 33722:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777497770 with bad export cookie 182717144640843868 [ 1031.216316] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1031.227200] LustreError: 33722:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1031.673541] Lustre: server umount lustre-MDT0001 complete [ 1042.700774] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 1045.559989] Key type lgssc unregistered [ 1046.000908] LNet: 36919:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1046.010850] LNetError: 36919:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1047.082093] LNet: Removed LNI 192.168.204.106@tcp [ 1048.070250] Key type .llcrypt unregistered [ 1048.073152] Key type ._llcrypt unregistered [ 1070.921468] Key type ._llcrypt registered [ 1070.923893] Key type .llcrypt registered [ 1071.043578] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 1072.414721] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1072.448071] alg: No test for adler32 (adler32-zlib) [ 1073.583336] Lustre: Lustre: Build Version: 2.17.52_93_g6cdb58a [ 1073.924361] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 1075.632183] Key type lgssc registered [ 1076.958459] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1085.128001] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1085.166610] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1086.815665] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1091.655025] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 1097.194357] LustreError: 38320:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1097.220948] LustreError: 38320:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 1100.253747] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1100.686760] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1105.658203] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 1111.446638] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1114.954117] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1121.532126] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1121.957959] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1127.778432] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 1132.248038] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1133.224870] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1141.187522] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.6@tcp (stopping) [ 1141.731900] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1141.753304] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1141.778285] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1142.754077] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1142.793532] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1142.814810] Lustre: Skipped 2 previous similar messages [ 1146.259139] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.6@tcp (stopping) [ 1146.587078] Lustre: server umount lustre-MDT0000 complete [ 1147.876046] LustreError: 39031:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1147.891272] LustreError: 39031:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 1150.739094] LustreError: 38299:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777497889 with bad export cookie 9483126781424272898 [ 1150.741291] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1150.750169] LustreError: 38299:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1151.386835] LustreError: 38316:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1151.415572] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.6@tcp (stopping) [ 1152.995535] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1153.013099] Lustre: Skipped 1 previous similar message [ 1156.001900] LustreError: 38314:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1156.021439] LustreError: 38314:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 1161.127782] LustreError: 40470:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1161.128751] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1161.156936] LustreError: 40470:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 1161.168974] Lustre: Skipped 3 previous similar messages [ 1165.280242] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1165.561222] Lustre: server umount lustre-MDT0001 complete [ 1180.570983] Lustre: server umount lustre-OST0000 complete [ 1197.123482] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 1200.135234] Key type lgssc unregistered [ 1200.442594] LNet: 41824:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1200.449952] LNetError: 41824:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1200.474741] LNet: Removed LNI 192.168.204.106@tcp [ 1201.290603] Key type .llcrypt unregistered [ 1201.293237] Key type ._llcrypt unregistered [ 1224.097429] Key type ._llcrypt registered [ 1224.103538] Key type .llcrypt registered [ 1224.241183] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 1225.479286] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1225.543194] alg: No test for adler32 (adler32-zlib) [ 1226.774784] Lustre: Lustre: Build Version: 2.17.52_93_g6cdb58a [ 1227.052912] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 1228.752402] Key type lgssc registered [ 1229.962259] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1237.295595] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1237.325713] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1238.736153] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1242.721951] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 1248.242708] LustreError: 43226:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1248.273712] LustreError: 43226:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 1250.455034] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1250.855174] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1255.435279] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 1260.874238] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1264.491948] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1271.190201] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1271.615653] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1278.406633] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 1283.852339] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1283.889221] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1291.235582] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1291.243184] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1291.253922] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1294.313733] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1294.320480] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1294.322743] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1294.349930] Lustre: Skipped 2 previous similar messages [ 1296.591443] Lustre: server umount lustre-OST0000 complete [ 1314.156335] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1314.198740] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1314.214829] Lustre: Skipped 1 previous similar message [ 1319.267872] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1319.271536] Lustre: Skipped 3 previous similar messages [ 1320.609798] Lustre: server umount lustre-MDT0000 complete [ 1324.010687] LustreError: 43206:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777498062 with bad export cookie 2680150501848669707 [ 1324.017091] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1324.020938] LustreError: 43206:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1338.336321] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1338.667629] Lustre: server umount lustre-MDT0001 complete [ 1348.768811] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 1351.483790] Key type lgssc unregistered [ 1351.845585] LNet: 46577:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1351.854552] LNetError: 46577:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1351.889733] LNet: Removed LNI 192.168.204.106@tcp [ 1352.825769] Key type .llcrypt unregistered [ 1352.830865] Key type ._llcrypt unregistered [ 1374.059414] Key type ._llcrypt registered [ 1374.062578] Key type .llcrypt registered [ 1374.170869] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 1375.213863] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1375.230181] alg: No test for adler32 (adler32-zlib) [ 1376.331978] Lustre: Lustre: Build Version: 2.17.52_93_g6cdb58a [ 1376.580599] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 1378.256951] Key type lgssc registered [ 1379.197634] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1385.172657] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1385.188639] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1386.629304] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1390.036524] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 1396.195763] LustreError: 47957:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1396.207624] LustreError: 47957:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 1397.054371] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1397.402292] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1400.780354] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 1405.391809] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1409.078830] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1414.824468] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1415.104416] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1420.056186] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 1424.706910] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1431.477349] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1440.704887] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1443.731180] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1455.728311] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-04-2917h29m52s [ 1457.220039] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1462.187656] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.6@tcp (stopping) [ 1462.242783] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1462.253693] Lustre: Skipped 2 previous similar messages [ 1463.256978] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.6@tcp (stopping) [ 1463.264299] Lustre: Skipped 3 previous similar messages [ 1465.041316] Lustre: server umount lustre-MDT0000 complete [ 1467.785152] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1468.311627] LustreError: 49023:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1469.413722] LustreError: 47957:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1469.429043] LustreError: 47957:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 1472.680347] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1472.795177] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1472.971955] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1473.007574] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1474.035159] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1476.454369] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 1479.606625] Lustre: DEBUG MARKER: Wait for df (36005) ... [ 1480.927653] Lustre: DEBUG MARKER: done [ 1488.354162] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1488.362612] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1488.375713] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1488.385228] Lustre: Skipped 3 previous similar messages [ 1488.868689] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1490.564727] Lustre: server umount lustre-OST0000 complete [ 1494.498244] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1494.499111] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1494.499975] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1494.499980] Lustre: Skipped 3 previous similar messages [ 1494.512846] Lustre: Skipped 2 previous similar messages [ 1499.656952] Lustre: server umount lustre-MDT0000 complete [ 1500.581029] LustreError: 47958:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1500.601464] LustreError: 47958:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 1502.903235] LustreError: 47938:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777498241 with bad export cookie 6531204306849652715 [ 1502.914583] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1502.915595] LustreError: 47938:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1503.246985] Lustre: server umount lustre-MDT0001 complete [ 1511.424395] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 1513.785795] Key type lgssc unregistered [ 1514.048995] LNet: 52524:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1514.059302] LNetError: 52524:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1514.079894] LNet: Removed LNI 192.168.204.106@tcp [ 1514.795729] Key type .llcrypt unregistered [ 1514.798314] Key type ._llcrypt unregistered [ 1546.014818] Key type ._llcrypt registered [ 1546.017797] Key type .llcrypt registered [ 1546.121770] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 1546.980778] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1547.003992] alg: No test for adler32 (adler32-zlib) [ 1548.018670] Lustre: Lustre: Build Version: 2.17.52_93_g6cdb58a [ 1548.215033] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 1549.872168] Key type lgssc registered [ 1550.844611] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1557.713187] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1557.745392] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1558.998135] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1559.024447] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1559.213455] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1562.852844] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 1569.716808] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1569.788153] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1569.807500] Lustre: 54695:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1569.990099] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1573.506572] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 1577.423666] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1580.108195] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1585.264693] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1585.481748] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1585.653120] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1591.127025] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 1591.400307] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1595.754260] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1601.010473] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-04-2917h32m18s [ 1602.121364] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1607.491359] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1609.261270] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1609.291993] LustreError: 54700:0:(mdt_handler.c:2347:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1629.619223] Lustre: lustre-MDT0000: Client b17ad1a4-9480-45c6-b31f-cc6c5f9664f0 (at 192.168.204.6@tcp) reconnecting [ 1649.378628] LustreError: 54700:0:(mdt_handler.c:2347:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1651.221953] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1653.075426] Lustre: DEBUG MARKER: done [ 1657.825852] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1657.827673] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1657.834610] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1657.846351] Lustre: Skipped 1 previous similar message [ 1657.859304] Lustre: Skipped 1 previous similar message [ 1662.950142] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1662.958484] Lustre: Skipped 1 previous similar message [ 1663.175416] Lustre: server umount lustre-OST0000 complete [ 1667.556172] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1667.567089] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1667.584300] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1672.357013] Lustre: server umount lustre-MDT0000 complete [ 1674.688586] LustreError: 57112:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777498413 with bad export cookie 906095144741132266 [ 1674.688996] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1674.695013] LustreError: 57112:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1674.860136] Lustre: server umount lustre-MDT0001 complete [ 1681.840398] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 1683.807435] Key type lgssc unregistered [ 1684.046719] LNet: 58665:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1684.054293] LNetError: 58665:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1684.072948] LNet: Removed LNI 192.168.204.106@tcp [ 1684.659510] Key type .llcrypt unregistered [ 1684.662260] Key type ._llcrypt unregistered [ 1714.411587] Key type ._llcrypt registered [ 1714.417713] Key type .llcrypt registered [ 1714.505872] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 1715.556055] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1715.589082] alg: No test for adler32 (adler32-zlib) [ 1716.652885] Lustre: Lustre: Build Version: 2.17.52_93_g6cdb58a [ 1716.849722] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 1718.480274] Key type lgssc registered [ 1719.381680] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1721.973969] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1721.997298] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1723.218601] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1723.243856] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1723.410407] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1724.194345] Lustre: Failing over lustre-MDT0000 [ 1724.459910] Lustre: server umount lustre-MDT0000 complete [ 1729.042566] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 17:34:26 (1777498466) [ 1733.949026] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1734.274967] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1737.087786] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 1743.103711] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1743.164945] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1743.190364] Lustre: 61427:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1743.372804] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1746.106983] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 1749.822376] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1752.213705] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1756.699287] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1756.831154] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1757.013087] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1759.402273] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1760.798645] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 1764.418190] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1768.536341] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1774.052153] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1774.057826] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1774.067619] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1775.050979] Lustre: server umount lustre-MDT0000 complete [ 1775.074374] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1777.236629] LustreError: 61419:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777498516 with bad export cookie 7215483296147311337 [ 1777.240935] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1777.248024] LustreError: 61419:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1777.479713] Lustre: server umount lustre-MDT0001 complete [ 1779.862995] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1783.024980] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1788.477489] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1796.320179] Lustre: 60350:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1777498519/real 1777498519] req@0000000073e86358 x1863842219984128/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1777498535 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1796.350186] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1796.361370] Lustre: Skipped 1 previous similar message [ 1798.624277] Lustre: 60351:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1777498521/real 1777498521] req@000000007338bc49 x1863842219984256/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1777498537 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1801.376138] Lustre: 60352:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1777498524/real 1777498524] req@00000000dfaa8f72 x1863842219984384/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1777498540 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1802.703627] LustreError: 63047:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1802.713655] LustreError: 63047:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 4 previous similar messages [ 1802.752122] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1802.806137] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1803.745346] Lustre: 60349:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1777498526/real 1777498526] req@00000000b3a80562 x1863842219984640/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1777498542 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1805.034931] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 1806.820276] LustreError: 64854:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1806.839851] LustreError: 64854:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 1808.864648] Lustre: 60349:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1777498531/real 1777498531] req@0000000070f6eb07 x1863842219984768/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1777498547 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1810.076137] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1812.499886] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 1815.414173] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1815.847570] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1818.061230] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1824.117743] Lustre: server umount lustre-MDT0000 complete [ 1825.769062] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1825.770768] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1825.775982] LustreError: 63048:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1825.779831] Lustre: Skipped 1 previous similar message [ 1826.548499] LustreError: 63994:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777498565 with bad export cookie 7215483296147313612 [ 1826.553696] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1826.556865] LustreError: 63994:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 1826.791141] Lustre: server umount lustre-MDT0001 complete [ 1831.015179] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1837.149477] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1846.240201] Lustre: 60349:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1777498569/real 1777498569] req@000000005ca7bb3e x1863842220005504/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1777498585 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1846.257581] Lustre: 60349:0:(client.c:2479:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1846.262738] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1846.278725] Lustre: Skipped 1 previous similar message [ 1851.427284] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x64228bd41da0192f [ 1851.432041] Lustre: MGC192.168.204.106@tcp: Connection restored to 0@lo (at 0@lo) [ 1851.437929] Lustre: Skipped 1 previous similar message [ 1851.582361] LustreError: 63048:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1851.595687] LustreError: 63048:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 1851.638903] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1851.644469] Lustre: Skipped 1 previous similar message [ 1851.680479] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 1854.129647] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 1856.741340] LustreError: 67308:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1856.753124] LustreError: 67308:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 1859.787312] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1863.122084] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 1865.198325] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1866.897878] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1869.193362] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1875.432256] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1875.432996] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1875.434300] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1875.449952] Lustre: Skipped 1 previous similar message [ 1879.368872] Lustre: server umount lustre-MDT0000 complete [ 1880.547978] LustreError: 67752:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1880.566893] LustreError: 67752:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 1881.675641] LustreError: 61419:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777498620 with bad export cookie 7215483296147314991 [ 1881.680160] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1881.681115] LustreError: 61419:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1881.895213] Lustre: server umount lustre-MDT0001 complete [ 1886.196740] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1888.587413] Lustre: DEBUG MARKER: files compared the same [ 1899.371725] Lustre: server umount lustre-OST0000 complete [ 1909.069623] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 1910.922494] Key type lgssc unregistered [ 1911.111565] LNet: 70335:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1911.120597] LNetError: 70335:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1911.134968] LNet: Removed LNI 192.168.204.106@tcp [ 1911.686372] Key type .llcrypt unregistered [ 1911.688100] Key type ._llcrypt unregistered [ 1927.513058] Key type ._llcrypt registered [ 1927.515802] Key type .llcrypt registered [ 1927.570585] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 1928.453698] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1928.480084] alg: No test for adler32 (adler32-zlib) [ 1929.497906] Lustre: Lustre: Build Version: 2.17.52_93_g6cdb58a [ 1929.669818] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 1931.328876] Key type lgssc registered [ 1932.374550] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1938.002474] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1938.025348] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1939.382636] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1941.971578] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug +malloc all [ 1947.503366] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1947.720721] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1950.324626] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug +malloc all [ 1953.790310] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1956.095818] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1960.232198] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1960.446878] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1964.176238] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug +malloc all [ 1964.531490] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 1967.625294] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1977.826031] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1977.827456] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1977.844071] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1977.848083] Lustre: Skipped 1 previous similar message [ 1979.818939] Lustre: server umount lustre-OST0000 complete [ 1982.945786] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1982.947173] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1982.958784] Lustre: Skipped 2 previous similar messages [ 1988.065485] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1988.072107] Lustre: Skipped 2 previous similar messages [ 1990.842326] Lustre: server umount lustre-MDT0000 complete [ 1992.973120] LustreError: 71694:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777498731 with bad export cookie 3530478537241384347 [ 1992.976074] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1992.984100] LustreError: 71694:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1993.234665] Lustre: server umount lustre-MDT0001 complete [ 1998.876705] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 2000.463717] Key type lgssc unregistered [ 2000.646413] LNet: 74884:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2000.653872] LNetError: 74884:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2000.670175] LNet: Removed LNI 192.168.204.106@tcp [ 2001.081543] Key type .llcrypt unregistered [ 2001.083212] Key type ._llcrypt unregistered [ 2015.603185] Key type ._llcrypt registered [ 2015.605186] Key type .llcrypt registered [ 2015.670702] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 2016.264981] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2016.337473] alg: No test for adler32 (adler32-zlib) [ 2017.308505] Lustre: Lustre: Build Version: 2.17.52_93_g6cdb58a [ 2017.449503] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 2019.048836] Key type lgssc registered [ 2019.773967] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2024.528699] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2024.541139] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2040.033241] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2045.153323] LustreError: 76234:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.106@tcp: failed processing log, type 4: rc = -110 [ 2071.777654] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2075.089558] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 2077.963678] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2082.139876] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2082.233417] Lustre: *** cfs_fail_loc=706, val=0*** [ 2082.458241] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2082.503185] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2084.345249] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 2088.413131] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2088.605082] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2090.483248] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 2093.129301] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2094.870975] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2098.146976] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2098.154617] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2098.161072] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2099.170957] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2099.181237] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2099.189764] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2102.019261] Lustre: server umount lustre-OST0000 complete [ 2109.410074] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2109.422757] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2109.437294] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2109.442217] Lustre: Skipped 1 previous similar message [ 2109.469714] Lustre: Skipped 2 previous similar messages [ 2111.401221] Lustre: server umount lustre-MDT0000 complete [ 2114.543193] LustreError: 77208:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2114.594204] LustreError: 77208:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 5 previous similar messages [ 2116.251374] LustreError: 76241:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777498855 with bad export cookie 4621389799917133344 [ 2116.265652] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2116.285953] LustreError: 76241:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2116.934778] Lustre: server umount lustre-MDT0001 complete [ 2128.253600] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 2130.672638] Key type lgssc unregistered [ 2131.003189] LNet: 79490:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2131.007899] LNetError: 79490:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2132.071734] LNet: Removed LNI 192.168.204.106@tcp [ 2132.891833] Key type .llcrypt unregistered [ 2132.894606] Key type ._llcrypt unregistered [ 2164.037948] Key type ._llcrypt registered [ 2164.043162] Key type .llcrypt registered [ 2164.417490] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 2167.478932] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2167.515435] alg: No test for adler32 (adler32-zlib) [ 2168.811331] Lustre: Lustre: Build Version: 2.17.52_93_g6cdb58a [ 2169.169899] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 2170.988366] Key type lgssc registered [ 2173.338611] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2188.838382] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2188.863602] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2196.347555] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 2205.305830] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2206.140529] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2211.278786] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 2211.304562] LustreError: 81562:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2211.332239] LustreError: 81562:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 2215.398866] LustreError: 81565:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2215.422740] LustreError: 81565:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 2220.519242] LustreError: 81566:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2220.532658] LustreError: 81566:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 2220.777260] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2221.252314] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2226.666768] LustreError: 81565:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2226.708200] LustreError: 81565:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 5 previous similar messages [ 2228.245479] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 2230.756138] LustreError: 81566:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2230.788706] LustreError: 81566:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 2236.945357] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2236.961248] Lustre: MGS: Client 38eab167-42e7-4061-a2aa-8c82ff3a1de8 (at 0@lo) reconnecting [ 2236.983777] LustreError: 81576:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2237.333858] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2237.465516] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2241.971697] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 2257.902501] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2257.903119] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2257.917789] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2257.957122] Lustre: Skipped 1 previous similar message [ 2262.050774] Lustre: server umount lustre-OST0000 complete [ 2268.139461] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2268.145689] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2268.171594] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2268.174127] Lustre: Skipped 1 previous similar message [ 2271.704808] Lustre: server umount lustre-MDT0000 complete [ 2273.253064] LustreError: 81565:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2273.279880] LustreError: 81565:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 4 previous similar messages [ 2275.597471] Lustre: server umount lustre-MDT0001 complete [ 2279.022496] Lustre: server umount MGS complete [ 2290.051118] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 17:43:47 (1777499027) [ 2320.656278] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_hostid [ 2329.986505] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 2364.129515] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2368.758367] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 2378.309402] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2378.542834] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2378.594450] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2378.702316] Lustre: lustre-MDT0001: new disk, initializing [ 2378.784125] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2378.808686] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2378.818381] Lustre: 88644:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2378.830341] LustreError: 88644:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2382.746441] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 2391.453633] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2391.689926] Lustre: lustre-OST0000: new disk, initializing [ 2391.694058] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2391.758221] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2397.272059] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 2401.830075] Lustre: 89851:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@00000000fb066f94 x1863842693948032/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1777499156 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2405.713626] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2412.903922] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2412.915954] Lustre: MGS: Client e8208bb9-b738-4c33-9494-cae8661d04b4 (at 0@lo) reconnecting [ 2412.984299] Lustre: 87928:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2413.023230] LustreError: 88630:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2413.047469] LustreError: 88630:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 2413.054659] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2413.160678] Lustre: lustre-MDT0000: new disk, initializing [ 2413.351230] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2413.420423] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2413.568418] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2414.497810] Lustre: 90349:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@00000000e3a51f7c x1863842693954432/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1777499169 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2418.395361] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 2419.633257] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2419.643565] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2419.780694] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2442.723397] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2442.741954] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2442.766926] Lustre: Skipped 1 previous similar message [ 2442.776963] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2444.775604] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2444.806380] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2444.818059] Lustre: Skipped 1 previous similar message [ 2448.851863] Lustre: server umount lustre-OST0000 complete [ 2455.031733] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2455.032652] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2455.059957] Lustre: Skipped 1 previous similar message [ 2455.062437] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2455.089150] Lustre: Skipped 1 previous similar message [ 2458.884865] Lustre: server umount lustre-MDT0000 complete [ 2460.132101] LustreError: 88619:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2460.161676] LustreError: 88619:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 2469.123795] Lustre: server umount lustre-MDT0001 complete [ 2473.364043] Lustre: server umount MGS complete [ 2485.424632] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 17:47:02 (1777499222) [ 2503.230865] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 2506.427674] Key type lgssc unregistered [ 2506.779204] LNet: 92814:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2506.789644] LNetError: 92814:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2506.810291] LNet: Removed LNI 192.168.204.106@tcp [ 2507.653888] Key type .llcrypt unregistered [ 2507.656606] Key type ._llcrypt unregistered [ 2526.647545] Key type ._llcrypt registered [ 2526.650298] Key type .llcrypt registered [ 2526.798968] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 2527.878949] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2527.987189] alg: No test for adler32 (adler32-zlib) [ 2529.291756] Lustre: Lustre: Build Version: 2.17.52_93_g6cdb58a [ 2529.561030] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 2531.272272] Key type lgssc registered [ 2532.405793] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2540.885287] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2540.907872] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2542.641258] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2547.634201] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 2552.806365] LustreError: 94019:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2552.831067] LustreError: 94019:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 2557.021900] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2557.575395] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2562.174356] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 2568.298833] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2571.987634] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2578.642435] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2579.017238] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2585.339952] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 2587.269624] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 2591.798316] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2602.977834] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2602.983797] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2602.993873] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2603.017740] Lustre: Skipped 1 previous similar message [ 2605.767075] Lustre: server umount lustre-OST0000 complete [ 2613.217517] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2613.240715] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2613.250960] Lustre: Skipped 2 previous similar messages [ 2615.710840] Lustre: server umount lustre-MDT0000 complete [ 2618.338990] LustreError: 94015:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2618.358565] LustreError: 94015:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 2619.604987] LustreError: 93999:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777499358 with bad export cookie 4246614640306584943 [ 2619.609675] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2619.620331] LustreError: 93999:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2620.196833] Lustre: server umount lustre-MDT0001 complete [ 2632.017796] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 2635.125268] Key type lgssc unregistered [ 2635.456274] LNet: 97196:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2635.465572] LNetError: 97196:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2635.483690] LNet: Removed LNI 192.168.204.106@tcp [ 2636.522763] Key type .llcrypt unregistered [ 2636.525612] Key type ._llcrypt unregistered [ 2653.148365] Key type ._llcrypt registered [ 2653.151536] Key type .llcrypt registered [ 2653.246672] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 2654.273104] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2654.381819] alg: No test for adler32 (adler32-zlib) [ 2655.489350] Lustre: Lustre: Build Version: 2.17.52_93_g6cdb58a [ 2655.787101] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 2657.496221] Key type lgssc registered [ 2658.748686] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2664.823447] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2664.828191] LustreError: 98264:0:(tgt_mount.c:2522:server_fill_super()) cfs_race id 716 sleeping [ 2664.833230] LustreError: 98263:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 waking [ 2664.838287] LustreError: 98264:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 awake: rc=4995 [ 2664.847605] LustreError: 98264:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 2664.855889] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2664.859400] LustreError: 98264:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2666.451874] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2674.135024] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2674.551460] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2679.313442] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 2685.051510] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing lsmod [ 2687.966185] LustreError: 99433:0:(tgt_mount.c:2522:server_fill_super()) cfs_race id 716 sleeping [ 2688.009980] LustreError: 99437:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 waking [ 2688.019848] LustreError: 99433:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 awake: rc=4955 [ 2688.032440] LustreError: 99433:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 2688.050514] LustreError: 99433:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2688.057928] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2688.472046] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2691.240502] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2691.244655] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2691.256145] Lustre: Skipped 1 previous similar message [ 2691.266974] Lustre: Skipped 2 previous similar messages [ 2696.681108] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2696.689673] Lustre: Skipped 2 previous similar messages [ 2697.569719] Lustre: server umount lustre-MDT0000 complete [ 2699.747938] LustreError: 98299:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2699.782535] LustreError: 98299:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 4 previous similar messages [ 2700.884902] LustreError: 98283:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777499439 with bad export cookie 8260135714335488805 [ 2700.889879] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2700.907570] LustreError: 98283:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2701.856391] LustreError: 98733:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000003729f647 x1863843204582784/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2701.983994] Lustre: server umount lustre-MDT0001 complete [ 2714.620371] Lustre: server umount lustre-OST0000 complete [ 2722.369395] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2722.888534] LustreError: 100499:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2722.988332] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2726.762026] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 2728.420042] LustreError: 100500:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2732.520257] LustreError: 100499:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2734.458970] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2734.841050] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2739.189358] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 2744.093619] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2747.208659] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2752.906726] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2753.220490] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2758.774398] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 2764.278803] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2767.550679] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 2775.009085] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2775.014574] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2775.025238] Lustre: Skipped 1 previous similar message [ 2775.029857] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2778.083696] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2778.084688] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2778.111232] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2780.666706] Lustre: server umount lustre-OST0000 complete [ 2786.277120] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2786.288339] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2786.304376] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2786.312896] Lustre: Skipped 1 previous similar message [ 2789.959902] Lustre: server umount lustre-MDT0000 complete [ 2793.435556] LustreError: 100480:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777499532 with bad export cookie 8260135714335489491 [ 2793.450367] LustreError: 100480:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2793.467062] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2793.762222] Lustre: server umount lustre-MDT0001 complete [ 2803.810908] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 2806.277963] Key type lgssc unregistered [ 2806.560240] LNet: 103674:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2806.567334] LNetError: 103674:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2806.593430] LNet: Removed LNI 192.168.204.106@tcp [ 2807.246509] Key type .llcrypt unregistered [ 2807.249654] Key type ._llcrypt unregistered [ 2827.909935] Key type ._llcrypt registered [ 2827.911698] Key type .llcrypt registered [ 2828.034716] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 2828.741562] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2828.803023] alg: No test for adler32 (adler32-zlib) [ 2829.912869] Lustre: Lustre: Build Version: 2.17.52_93_g6cdb58a [ 2830.198864] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 2831.840592] Key type lgssc registered [ 2833.015444] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2840.279899] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2840.293287] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2841.685498] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2845.017439] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 2851.814967] LustreError: 105055:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2851.832148] LustreError: 105055:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 2852.277481] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2852.560887] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2856.871633] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 2861.987643] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2865.391674] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2871.084404] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2871.383479] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2876.182404] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 2880.934723] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2886.769703] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 2890.273945] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 2892.766802] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 2897.387964] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2897.404363] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2897.404852] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2897.424671] Lustre: Skipped 1 previous similar message [ 2901.811406] Lustre: server umount lustre-OST0000 complete [ 2907.624532] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2907.626202] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2907.636413] Lustre: Skipped 2 previous similar messages [ 2911.338543] Lustre: server umount lustre-MDT0000 complete [ 2912.738574] LustreError: 106067:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2912.752756] LustreError: 106067:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 2914.286396] LustreError: 105037:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777499653 with bad export cookie 18313621936633281013 [ 2914.291473] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2914.300168] LustreError: 105037:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2914.682467] Lustre: server umount lustre-MDT0001 complete [ 2924.079116] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 2926.646981] Key type lgssc unregistered [ 2927.026332] LNet: 108334:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2927.034498] LNetError: 108334:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2927.056936] LNet: Removed LNI 192.168.204.106@tcp [ 2927.775968] Key type .llcrypt unregistered [ 2927.777857] Key type ._llcrypt unregistered [ 2943.522673] Key type ._llcrypt registered [ 2943.526938] Key type .llcrypt registered [ 2943.616193] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 2944.779103] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2944.826497] alg: No test for adler32 (adler32-zlib) [ 2945.914571] Lustre: Lustre: Build Version: 2.17.52_93_g6cdb58a [ 2946.184085] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 2947.889298] Key type lgssc registered [ 2949.029333] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2956.207479] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2956.226271] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2957.851612] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2961.838466] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 2968.035649] LustreError: 109542:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2968.058465] LustreError: 109542:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 2969.337348] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2969.636128] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2973.454352] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 2978.587723] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2982.078900] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2987.827970] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2988.133390] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2993.481341] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 2998.490762] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3002.484491] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3013.092657] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3013.095356] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3013.097132] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3013.111028] Lustre: Skipped 1 previous similar message [ 3013.132718] Lustre: Skipped 1 previous similar message [ 3016.711594] Lustre: server umount lustre-OST0000 complete [ 3020.771295] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3020.775058] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3020.803844] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3023.335902] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3023.338507] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3025.989869] Lustre: server umount lustre-MDT0000 complete [ 3028.449764] LustreError: 110264:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3028.481098] LustreError: 110264:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 3029.179239] LustreError: 109523:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777499767 with bad export cookie 4778290594051842391 [ 3029.181987] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3029.189477] LustreError: 109523:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3029.484087] Lustre: server umount lustre-MDT0001 complete [ 3037.159477] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 3039.554848] Key type lgssc unregistered [ 3039.779814] LNet: 112737:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3039.786501] LNetError: 112737:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3039.801792] LNet: Removed LNI 192.168.204.106@tcp [ 3040.433460] Key type .llcrypt unregistered [ 3040.437553] Key type ._llcrypt unregistered [ 3057.915056] Key type ._llcrypt registered [ 3057.926749] Key type .llcrypt registered [ 3058.008871] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 3058.857944] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3058.929993] alg: No test for adler32 (adler32-zlib) [ 3059.892730] Lustre: Lustre: Build Version: 2.17.52_93_g6cdb58a [ 3060.056366] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 3061.672717] Key type lgssc registered [ 3062.751120] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3068.795507] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3068.826510] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3070.186796] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3073.434124] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 3079.653291] LustreError: 114118:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3079.662217] LustreError: 114118:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 3080.126147] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3080.433812] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3083.770990] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 3087.911570] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3090.464790] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3095.524494] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3095.841877] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3100.985407] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 3105.630562] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3114.286584] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3120.698389] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3123.542297] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3123.551678] Lustre: Skipped 2 previous similar messages [ 3126.828312] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3126.833939] Lustre: Skipped 2 previous similar messages [ 3128.383674] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3128.408524] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3133.536570] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.204.6@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3133.543336] Lustre: Skipped 2 previous similar messages [ 3138.588856] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.204.6@tcp 192.168.0.[2,10]@tcp [ 3141.834460] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.204.6/24@tcp in log lustre-MDT0000 [ 3141.841506] Lustre: Skipped 2 previous similar messages [ 3146.805226] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.6/24@tcp [ 3146.811445] Lustre: Skipped 1 previous similar message [ 3152.355298] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3152.358906] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3152.367978] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3155.427967] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3155.429937] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3155.430747] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3155.434890] Lustre: Skipped 1 previous similar message [ 3157.821781] Lustre: server umount lustre-OST0000 complete [ 3162.594670] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3162.606695] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3162.623728] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3165.667662] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3165.671723] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3165.684473] Lustre: Skipped 1 previous similar message [ 3170.787669] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3170.793062] Lustre: Skipped 1 previous similar message [ 3174.882251] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3175.116852] Lustre: server umount lustre-MDT0000 complete [ 3175.908193] LustreError: 114117:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3175.944760] LustreError: 114117:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 3178.191586] LustreError: 114098:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777499917 with bad export cookie 3412309615827633390 [ 3178.193933] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3178.197848] LustreError: 114098:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3178.484422] Lustre: server umount lustre-MDT0001 complete [ 3186.244344] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 3188.493983] Key type lgssc unregistered [ 3188.710393] LNet: 118672:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3188.716750] LNetError: 118672:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3188.731856] LNet: Removed LNI 192.168.204.106@tcp [ 3189.386403] Key type .llcrypt unregistered [ 3189.388643] Key type ._llcrypt unregistered [ 3208.928180] Key type ._llcrypt registered [ 3208.934218] Key type .llcrypt registered [ 3209.004155] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 3209.781922] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3209.808965] alg: No test for adler32 (adler32-zlib) [ 3210.828618] Lustre: Lustre: Build Version: 2.17.52_93_g6cdb58a [ 3211.047102] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 3212.683725] Key type lgssc registered [ 3213.581472] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3219.772430] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3219.802321] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3221.180498] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3221.211893] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.204.6/24@tcp [ 3221.212688] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3224.689281] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 3230.694989] LustreError: 120195:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3230.706334] LustreError: 120195:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 3231.390975] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3231.618136] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3231.640326] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.6/24@tcp [ 3231.674338] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3234.804742] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 3238.963181] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3241.854809] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3246.466393] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3246.749873] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3251.567771] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 3255.671916] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3265.208161] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 3275.748507] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3275.750976] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3275.755898] Lustre: Skipped 1 previous similar message [ 3279.380512] Lustre: server umount lustre-OST0000 complete [ 3282.914112] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3282.921463] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3282.932503] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3282.937591] Lustre: Skipped 1 previous similar message [ 3285.988497] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3285.988906] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3288.385800] Lustre: server umount lustre-MDT0000 complete [ 3290.980395] LustreError: 120174:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777500029 with bad export cookie 12786267358050618913 [ 3290.988258] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3290.996415] LustreError: 120174:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3291.232373] Lustre: server umount lustre-MDT0001 complete [ 3298.522897] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 3300.749028] Key type lgssc unregistered [ 3301.001710] LNet: 123446:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3301.012557] LNetError: 123446:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3301.024133] LNet: Removed LNI 192.168.204.106@tcp [ 3301.620778] Key type .llcrypt unregistered [ 3301.622510] Key type ._llcrypt unregistered [ 3320.051410] Key type ._llcrypt registered [ 3320.055389] Key type .llcrypt registered [ 3320.138724] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_hostid [ 3330.712332] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 3331.397827] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3331.414815] alg: No test for adler32 (adler32-zlib) [ 3332.366407] Lustre: Lustre: Build Version: 2.17.52_93_g6cdb58a [ 3332.541226] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 3334.168350] Key type lgssc registered [ 3335.055728] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3357.675562] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3357.703082] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3358.848150] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3358.870798] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3358.943443] Lustre: lustre-MDT0000: new disk, initializing [ 3359.013406] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3359.030037] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3361.758107] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 3368.914776] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3369.000943] Lustre: 126802:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3369.051283] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3369.055562] Lustre: Skipped 1 previous similar message [ 3369.113255] Lustre: lustre-MDT0001: new disk, initializing [ 3369.188261] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3369.209479] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3369.216363] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3372.246481] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 3377.427996] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3379.784237] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3384.341443] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3384.564607] Lustre: lustre-OST0000: new disk, initializing [ 3384.567605] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3384.627362] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3388.546877] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 3389.979834] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3389.985673] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3390.062798] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3393.434750] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3396.190297] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3396.323861] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3398.959528] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3399.105454] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3405.286306] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3405.286341] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3405.289475] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3405.292438] LustreError: Skipped 1 previous similar message [ 3405.309483] Lustre: Skipped 1 previous similar message [ 3406.391953] Lustre: server umount lustre-OST0000 complete [ 3410.402820] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3410.403170] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3410.408517] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3410.408525] Lustre: Skipped 1 previous similar message [ 3410.413820] Lustre: Skipped 1 previous similar message [ 3414.792058] Lustre: server umount lustre-MDT0000 complete [ 3416.867451] LustreError: 126795:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777500155 with bad export cookie 5872845956535802520 [ 3416.867965] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3416.876029] LustreError: 126795:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3417.079484] Lustre: server umount lustre-MDT0001 complete [ 3422.047227] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3422.319337] LustreError: 130608:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3422.329766] LustreError: 130608:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 3422.364252] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3424.693259] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 3427.809806] LustreError: 130609:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3429.721805] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3429.925449] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3432.449244] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 3435.562776] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3437.852815] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3442.122603] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3442.377919] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3445.769049] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 3449.021716] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3451.426292] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3451.544751] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3453.772209] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3454.924208] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 1 sec [ 3461.312971] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3461.431181] Lustre: lustre-OST0001: new disk, initializing [ 3461.435663] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3461.506662] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3464.856603] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 3469.504688] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3471.399974] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3471.407985] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3471.480856] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3471.748268] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 3474.217552] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3474.344943] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3476.679110] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff99f105155800.ost_server_uuid 50 [ 3477.566550] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff99f105155800.ost_server_uuid in FULL state after 0 sec [ 3486.690656] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3486.693150] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3486.699727] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3486.708484] Lustre: Skipped 1 previous similar message [ 3486.715452] Lustre: Skipped 2 previous similar messages [ 3488.228509] Lustre: server umount lustre-OST0001 complete [ 3490.786428] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3490.795225] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3490.807221] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3491.812181] LustreError: 132199:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3491.822432] LustreError: 132199:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 3496.377884] Lustre: server umount lustre-OST0000 complete [ 3501.537840] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3501.543508] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3501.553450] Lustre: Skipped 1 previous similar message [ 3501.557483] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3501.561077] Lustre: Skipped 2 previous similar messages [ 3504.726439] Lustre: server umount lustre-MDT0000 complete [ 3507.173435] LustreError: 130608:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3507.195573] LustreError: 130608:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 3507.209937] LustreError: 133608:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777500246 with bad export cookie 5872845956535803402 [ 3507.213475] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3507.217261] LustreError: 133608:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 3507.464906] Lustre: server umount lustre-MDT0001 complete [ 3518.033864] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 3519.496622] Key type lgssc unregistered [ 3519.647746] LNet: 136460:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3519.651623] LNetError: 136460:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3519.661916] LNet: Removed LNI 192.168.204.106@tcp [ 3520.085220] Key type .llcrypt unregistered [ 3520.088406] Key type ._llcrypt unregistered [ 3542.146762] Key type ._llcrypt registered [ 3542.148494] Key type .llcrypt registered [ 3542.216781] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_hostid [ 3550.833427] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 3551.334470] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3551.410770] alg: No test for adler32 (adler32-zlib) [ 3552.329762] Lustre: Lustre: Build Version: 2.17.52_93_g6cdb58a [ 3552.485973] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 3554.112967] Key type lgssc registered [ 3554.864337] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3575.243459] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3575.262251] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3576.440809] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3576.462805] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3576.518943] Lustre: lustre-MDT0000: new disk, initializing [ 3576.583397] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3576.604900] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3578.884748] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 3585.590770] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3585.655550] Lustre: 140553:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3585.696940] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3585.700587] Lustre: Skipped 1 previous similar message [ 3585.753407] Lustre: lustre-MDT0001: new disk, initializing [ 3585.827970] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3585.856494] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3585.863292] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3588.269665] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 3592.871567] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3594.928563] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3598.855555] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3599.068613] Lustre: lustre-OST0000: new disk, initializing [ 3599.072276] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3599.135602] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3602.074557] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 3606.609421] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3607.101151] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3607.109952] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3607.166970] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3610.305975] Lustre: Failing over lustre-OST0000 [ 3610.413990] Lustre: server umount lustre-OST0000 complete [ 3624.816710] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3624.983611] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3624.989868] Lustre: Skipped 1 previous similar message [ 3625.002532] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3625.015517] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 3628.064273] Lustre: 138780:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1777500350/real 1777500350] req@00000000a21b9c8e x1863844145163776/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1777500366 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3628.080481] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 3629.024144] Lustre: 138783:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1777500351/real 1777500351] req@00000000ac037537 x1863844145164032/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1777500367 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3629.037385] Lustre: 138783:0:(client.c:2479:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 3629.474046] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 3630.071884] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3630.072062] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 3630.087386] Lustre: Failing over lustre-MDT0000 [ 3630.211721] Lustre: server umount lustre-MDT0000 complete [ 3632.097142] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3632.100047] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3633.185414] Lustre: 138780:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1777500356/real 1777500356] req@00000000f2b4261d x1863844145165184/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1777500372 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3633.205967] Lustre: 138780:0:(client.c:2479:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 3635.169827] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3635.181672] Lustre: Skipped 1 previous similar message [ 3638.368150] Lustre: 138780:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1777500361/real 1777500361] req@00000000dcad7503 x1863844145166336/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1777500377 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3639.698225] LustreError: 140560:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3639.706479] LustreError: 140560:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 6 previous similar messages [ 3640.290642] LustreError: 140566:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3640.301649] LustreError: 140566:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 3644.818294] LustreError: 140561:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3645.005674] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3645.087144] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3645.093940] Lustre: 140557:0:(mgc_request.c:1897:mgc_process_log()) MGC192.168.204.106@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 3645.265980] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3645.302417] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3647.470184] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 3649.970480] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3650.534756] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3650.538852] Lustre: Skipped 1 previous similar message [ 3650.548019] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 3650.569581] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 3655.650733] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3655.651074] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3655.654898] LustreError: Skipped 1 previous similar message [ 3655.655869] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3655.662377] Lustre: Skipped 1 previous similar message [ 3658.418298] Lustre: server umount lustre-OST0000 complete [ 3660.770140] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3660.771216] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3660.772934] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3660.772940] Lustre: Skipped 1 previous similar message [ 3660.778890] Lustre: Skipped 1 previous similar message [ 3665.889269] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3665.893181] Lustre: Skipped 2 previous similar messages [ 3666.511469] Lustre: server umount lustre-MDT0000 complete [ 3668.603370] LustreError: 140547:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777500407 with bad export cookie 4711396132570352393 [ 3668.604710] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3668.615458] LustreError: 140547:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3668.805278] Lustre: server umount lustre-MDT0001 complete [ 3674.416940] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 3676.012825] Key type lgssc unregistered [ 3676.174506] LNet: 145779:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3676.181337] LNetError: 145779:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3677.221741] LNet: Removed LNI 192.168.204.106@tcp [ 3677.701173] Key type .llcrypt unregistered [ 3677.703647] Key type ._llcrypt unregistered [ 3702.327861] Key type ._llcrypt registered [ 3702.329785] Key type .llcrypt registered [ 3702.389154] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 3702.948250] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3702.998642] alg: No test for adler32 (adler32-zlib) [ 3703.920865] Lustre: Lustre: Build Version: 2.17.52_93_g6cdb58a [ 3704.052325] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 3705.672615] Key type lgssc registered [ 3706.341789] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3710.639452] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3710.654720] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3711.939214] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3714.260730] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 3718.842358] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3721.187098] LustreError: 147281:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3721.205816] LustreError: 147281:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 3721.544772] LustreError: 147309:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 3721.550963] LustreError: 147309:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 3721.555932] LustreError: 147309:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 10, retries 0, failed: rc = -5 [ 3721.742023] Lustre: server umount lustre-MDT0000 complete [ 3729.611943] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_hostid [ 3734.065801] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 3752.444467] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3752.584255] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3752.605993] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3752.656240] Lustre: lustre-MDT0000: new disk, initializing [ 3752.707624] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3752.720965] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3754.639957] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 3760.519252] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3760.580687] Lustre: 150459:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3760.617032] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3760.620109] Lustre: Skipped 1 previous similar message [ 3760.667254] Lustre: lustre-MDT0001: new disk, initializing [ 3760.722565] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3760.742721] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3760.749462] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3762.624919] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 3766.954855] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3768.720923] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3772.373727] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3772.545463] Lustre: lustre-OST0000: new disk, initializing [ 3772.548605] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3772.608401] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3773.325571] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3773.334066] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3773.384880] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3775.774401] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 3780.189813] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3783.672602] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 3784.722879] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 18:08:42 (1777500522) [ 3924.448867] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3924.455556] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3924.465524] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3927.010314] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3927.012568] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3928.853729] Lustre: server umount lustre-MDT0000 complete [ 3930.616361] LustreError: 150449:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777500669 with bad export cookie 4980056102450847053 [ 3930.616879] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3930.622704] LustreError: 150449:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3932.129309] LustreError: 150466:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3932.129643] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3932.150936] Lustre: Skipped 1 previous similar message [ 3932.154099] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3932.156868] Lustre: Skipped 2 previous similar messages [ 3935.140374] LustreError: 152205:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3935.141638] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3936.823421] Lustre: server umount lustre-MDT0001 complete [ 3953.120112] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3953.199616] Lustre: server umount lustre-OST0000 complete [ 3957.888501] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_hostid [ 3963.038091] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 3981.742229] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3981.861738] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3981.881970] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3981.932648] Lustre: lustre-MDT0000: new disk, initializing [ 3981.981944] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3981.996174] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3983.840907] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 3989.285454] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3989.338631] Lustre: 156689:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3989.370193] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3989.373680] Lustre: Skipped 1 previous similar message [ 3989.422232] Lustre: lustre-MDT0001: new disk, initializing [ 3989.479603] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3989.485103] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3991.179831] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 3995.276144] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3997.187644] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4000.547996] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4000.688363] Lustre: lustre-OST0000: new disk, initializing [ 4000.690429] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4000.735427] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4000.741097] Lustre: Skipped 1 previous similar message [ 4003.013051] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4003.017522] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4003.075595] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4003.589796] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4007.981487] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4009.990268] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4010.095590] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4012.133386] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4012.256454] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4013.537841] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4013.539266] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4013.547975] LustreError: Skipped 1 previous similar message [ 4013.551441] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4013.562257] Lustre: Skipped 1 previous similar message [ 4013.573635] Lustre: Skipped 1 previous similar message [ 4019.443813] Lustre: server umount lustre-OST0000 complete [ 4023.778586] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4023.779243] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4023.784801] Lustre: Skipped 1 previous similar message [ 4023.790134] Lustre: Skipped 3 previous similar messages [ 4027.441826] Lustre: server umount lustre-MDT0000 complete [ 4028.897595] LustreError: 157472:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4028.909789] LustreError: 157472:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 4029.170393] LustreError: 156684:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777500767 with bad export cookie 4980056102451005883 [ 4029.172983] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4029.175447] LustreError: 156684:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4029.332305] Lustre: server umount lustre-MDT0001 complete [ 4033.613307] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 18:12:51 (1777500771) [ 4042.244570] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_hostid [ 4047.200754] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 4063.592057] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4063.727525] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4063.751573] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4063.798861] Lustre: lustre-MDT0000: new disk, initializing [ 4063.852350] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4063.862978] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4065.687434] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4071.428393] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4071.487377] Lustre: 162999:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4071.614958] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4073.551967] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4077.762188] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4079.707629] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4083.136126] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4085.916810] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4089.909850] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4091.922689] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4091.973414] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4102.115267] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4102.117834] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4102.131041] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4102.133371] Lustre: Skipped 1 previous similar message [ 4104.552830] Lustre: server umount lustre-OST0000 complete [ 4112.495468] Lustre: server umount lustre-MDT0000 complete [ 4114.287439] LustreError: 162993:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777500853 with bad export cookie 4980056102451006765 [ 4114.289760] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4114.293532] LustreError: 162993:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4114.458392] Lustre: server umount lustre-MDT0001 complete [ 4118.145144] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 18:14:15 (1777500855) [ 4124.958397] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_hostid [ 4129.415690] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 4145.088156] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4145.201349] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4145.215715] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4145.218551] Lustre: Skipped 3 previous similar messages [ 4145.261435] Lustre: lustre-MDT0000: new disk, initializing [ 4145.263715] Lustre: Skipped 2 previous similar messages [ 4145.305033] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4145.307703] Lustre: Skipped 2 previous similar messages [ 4145.316113] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4145.320594] Lustre: Skipped 2 previous similar messages [ 4147.010892] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4152.379202] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4152.438206] Lustre: 168920:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4152.593788] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4154.572461] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4158.462755] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4160.105548] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4163.155976] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4165.807648] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4169.566058] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4172.337257] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4182.500254] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4182.501095] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4182.503045] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4182.503052] Lustre: Skipped 4 previous similar messages [ 4182.507071] Lustre: Skipped 4 previous similar messages [ 4182.510574] LustreError: Skipped 2 previous similar messages [ 4183.645196] Lustre: server umount lustre-OST0000 complete [ 4191.415126] Lustre: server umount lustre-MDT0000 complete [ 4192.738468] LustreError: 169818:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4193.083430] LustreError: 169700:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777500931 with bad export cookie 4980056102451007829 [ 4193.087159] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4193.091149] LustreError: 169700:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4193.259730] Lustre: server umount lustre-MDT0001 complete [ 4198.216028] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 4199.758314] Key type lgssc unregistered [ 4199.938782] LNet: 172392:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4199.944392] LNetError: 172392:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4199.955721] LNet: Removed LNI 192.168.204.106@tcp [ 4200.392412] Key type .llcrypt unregistered [ 4200.394362] Key type ._llcrypt unregistered [ 4211.821970] Key type ._llcrypt registered [ 4211.823489] Key type .llcrypt registered [ 4211.896656] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 4212.430572] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4212.439808] alg: No test for adler32 (adler32-zlib) [ 4213.334684] Lustre: Lustre: Build Version: 2.17.52_93_g6cdb58a [ 4213.444456] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 4215.048858] Key type lgssc registered [ 4215.570521] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4219.378432] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4219.390272] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4220.632958] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4222.728094] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4226.766196] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4226.962422] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4228.939370] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4231.671902] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4233.553874] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4236.855394] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4237.050237] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4239.897922] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4242.477794] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4245.223212] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4266.977927] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4266.981335] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4266.988775] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4266.993555] Lustre: Skipped 1 previous similar message [ 4272.099381] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4272.106140] Lustre: Skipped 1 previous similar message [ 4272.298896] Lustre: server umount lustre-OST0000 complete [ 4277.218587] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4277.224963] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4277.227603] Lustre: Skipped 2 previous similar messages [ 4277.235410] Lustre: Skipped 1 previous similar message [ 4280.505807] Lustre: server umount lustre-MDT0000 complete [ 4282.338162] LustreError: 173766:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4282.345251] LustreError: 173766:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 4282.398835] LustreError: 173748:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777501021 with bad export cookie 10909600991063622945 [ 4282.399581] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4282.405171] LustreError: 173748:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4282.554302] Lustre: server umount lustre-MDT0001 complete [ 4287.361963] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 4288.735941] Key type lgssc unregistered [ 4288.888519] LNet: 176964:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4288.892550] LNetError: 176964:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4288.904183] LNet: Removed LNI 192.168.204.106@tcp [ 4289.323760] Key type .llcrypt unregistered [ 4289.327941] Key type ._llcrypt unregistered [ 4301.390599] Key type ._llcrypt registered [ 4301.392583] Key type .llcrypt registered [ 4301.458423] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 4301.997519] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4302.018253] alg: No test for adler32 (adler32-zlib) [ 4302.903856] Lustre: Lustre: Build Version: 2.17.52_93_g6cdb58a [ 4303.041320] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 4304.664204] Key type lgssc registered [ 4305.315474] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4309.359036] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4309.379261] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4310.646399] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4312.620766] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4316.130965] LustreError: 178342:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4316.140788] LustreError: 178342:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 4317.416173] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4317.682948] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4319.767195] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4322.775417] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4324.832967] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4328.614110] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4328.822084] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4332.185940] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4335.110560] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4335.984470] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 4348.388111] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4348.388229] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4348.389123] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4348.394876] Lustre: Skipped 1 previous similar message [ 4351.901479] Lustre: lustre-OST0000: Not available for connect from 192.168.204.6@tcp (stopping) [ 4351.905206] Lustre: Skipped 1 previous similar message [ 4353.507940] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4353.514021] Lustre: Skipped 1 previous similar message [ 4354.279889] Lustre: server umount lustre-OST0000 complete [ 4357.404726] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff99f110a09800.ost_server_uuid 50 [ 4358.364435] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 4373.986428] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4373.988791] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4373.997238] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4374.006046] Lustre: Skipped 1 previous similar message [ 4375.718967] Lustre: server umount lustre-MDT0000 complete [ 4377.344795] LustreError: 178322:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777501116 with bad export cookie 14073438862110002647 [ 4377.347804] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4377.351197] LustreError: 178322:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4377.485970] Lustre: server umount lustre-MDT0001 complete [ 4381.072164] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 18:18:38 (1777501118) [ 4383.976048] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4384.206322] LustreError: 181831:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4384.246753] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4386.182222] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4389.347146] LustreError: 181832:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4389.971825] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4390.173171] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4392.113296] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4394.710899] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4396.783708] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4400.301746] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4400.511140] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4403.513450] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4406.227844] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4409.710748] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 4410.034900] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4410.163157] Lustre: lustre-OST0001: new disk, initializing [ 4410.166629] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4413.703325] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4417.568658] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4417.575317] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4417.618407] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4418.135506] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4419.554621] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4419.558680] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4419.570349] Lustre: Skipped 1 previous similar message [ 4419.572888] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4425.612390] Lustre: server umount lustre-OST0000 complete [ 4427.748566] LustreError: 183425:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4427.756312] LustreError: 183425:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 4428.099895] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4428.193774] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 4429.975587] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4430.088779] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 4432.870671] LustreError: 183423:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4432.878705] LustreError: 183423:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 4443.108035] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4443.110714] LustreError: 183425:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4443.111460] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4443.111465] LustreError: Skipped 1 previous similar message [ 4443.112851] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4443.112855] Lustre: Skipped 4 previous similar messages [ 4443.118992] Lustre: Skipped 2 previous similar messages [ 4443.124000] LustreError: 183425:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 5 previous similar messages [ 4446.884266] Lustre: server umount lustre-OST0001 complete [ 4451.809105] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4451.812535] LustreError: Skipped 1 previous similar message [ 4451.815123] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4463.073516] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4463.146498] Lustre: server umount lustre-MDT0000 complete [ 4463.585856] LustreError: 181831:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4463.594361] LustreError: 181831:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 4464.802625] LustreError: 181813:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777501203 with bad export cookie 14073438862110003704 [ 4464.806593] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4464.810506] LustreError: 181813:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4464.960756] Lustre: server umount lustre-MDT0001 complete [ 4474.870961] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 18:20:12 (1777501212) [ 4477.623786] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4477.753592] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4477.771573] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4477.883109] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4477.887693] Lustre: Skipped 1 previous similar message [ 4479.812860] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4484.102525] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4484.167893] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4484.191987] Lustre: 187593:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4486.244250] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4488.744763] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4490.617573] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4493.680511] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4493.776211] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4496.238371] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4498.515908] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4498.971596] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 4501.481048] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4501.519526] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 4503.962723] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4506.258767] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4508.151218] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4508.153041] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4508.158451] Lustre: Skipped 2 previous similar messages [ 4508.163080] Lustre: Skipped 7 previous similar messages [ 4508.180830] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 4513.483234] Lustre: server umount lustre-OST0000 complete [ 4516.753361] LustreError: 190778:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4528.612759] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4528.615561] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4528.621441] LustreError: Skipped 1 previous similar message [ 4531.172819] Lustre: server umount lustre-OST0001 complete [ 4543.969051] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4543.971591] Lustre: Skipped 10 previous similar messages [ 4547.040184] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4547.114775] Lustre: server umount lustre-MDT0000 complete [ 4548.618296] LustreError: 187587:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777501287 with bad export cookie 14073438862110005090 [ 4548.621388] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4548.625661] LustreError: 187587:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4548.755278] Lustre: server umount lustre-MDT0001 complete [ 4558.382316] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 18:21:36 (1777501296) [ 4564.782895] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_hostid [ 4568.729393] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 4583.903307] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4584.010683] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4584.027284] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4584.072532] Lustre: lustre-MDT0000: new disk, initializing [ 4584.111912] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4584.116036] Lustre: Skipped 3 previous similar messages [ 4584.124124] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4585.825168] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4591.092726] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4591.142273] Lustre: 195171:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4591.175838] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4591.179572] Lustre: Skipped 1 previous similar message [ 4591.220918] Lustre: lustre-MDT0001: new disk, initializing [ 4591.282889] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4591.290614] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4592.922363] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4596.946439] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4598.704575] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4601.729245] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4601.849179] Lustre: lustre-OST0000: new disk, initializing [ 4601.851822] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4604.320989] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4608.100810] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4609.542352] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4609.548045] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4609.605489] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4610.058540] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4610.205995] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4612.285429] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4612.401492] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4614.625414] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4614.629047] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4614.630456] LustreError: Skipped 1 previous similar message [ 4614.637688] Lustre: Skipped 3 previous similar messages [ 4614.640786] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4614.646968] Lustre: Skipped 1 previous similar message [ 4615.460942] Lustre: server umount lustre-OST0000 complete [ 4623.420554] Lustre: server umount lustre-MDT0000 complete [ 4624.865467] LustreError: 195964:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4624.875324] LustreError: 195964:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 8 previous similar messages [ 4625.057846] LustreError: 195163:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777501363 with bad export cookie 14073438862110006847 [ 4625.058793] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4625.068352] LustreError: 195163:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4625.208782] Lustre: server umount lustre-MDT0001 complete [ 4628.899336] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4630.654783] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4634.176778] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4636.044087] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4638.397079] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4639.826991] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4642.740201] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4645.414723] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4647.720876] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4649.430461] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4654.670516] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 5 sec [ 4656.404716] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4656.524863] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4659.683879] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4663.526946] Lustre: server umount lustre-OST0000 complete [ 4666.090609] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4667.884117] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4692.409396] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4692.569684] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4692.574793] Lustre: Skipped 5 previous similar messages [ 4695.030396] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4697.639133] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4698.407600] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4699.778753] Lustre: server umount lustre-OST0000 complete [ 4706.274390] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4706.282696] Lustre: Skipped 5 previous similar messages [ 4707.691874] Lustre: server umount lustre-MDT0000 complete [ 4709.359918] LustreError: 203578:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777501448 with bad export cookie 14073438862110007729 [ 4709.364546] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4709.368110] LustreError: 203578:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4709.515142] Lustre: server umount lustre-MDT0001 complete [ 4713.166133] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 18:24:10 (1777501450) [ 4715.796059] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4717.517227] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4720.937355] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4722.716679] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4724.891183] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4726.396192] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4729.234534] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4731.613893] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4733.666170] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4735.200587] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4741.430575] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 6 sec [ 4743.028945] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4743.118574] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4745.843030] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4745.921564] Lustre: lustre-OST0001: new disk, initializing [ 4745.924876] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4748.239451] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4751.596514] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4752.405674] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4752.410833] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4752.445661] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4753.196374] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4753.293892] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4754.999562] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4755.091225] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4757.475390] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4757.480421] LustreError: Skipped 2 previous similar messages [ 4757.480604] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4757.489084] Lustre: Skipped 7 previous similar messages [ 4761.994820] Lustre: server umount lustre-OST0001 complete [ 4762.599508] LustreError: 206040:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4762.605728] LustreError: 206040:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 4764.198442] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4764.292297] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 4765.713648] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4765.803939] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 4789.811688] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4791.971429] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4794.142833] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4794.840487] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4795.367934] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 4795.372830] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4795.372895] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 4801.795179] Lustre: server umount lustre-OST0001 complete [ 4820.254196] Lustre: server umount lustre-OST0000 complete [ 4827.868408] Lustre: server umount lustre-MDT0000 complete [ 4829.245181] LustreError: 207484:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777501568 with bad export cookie 14073438862110008660 [ 4829.248578] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4829.250165] LustreError: 207484:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4835.455105] Lustre: server umount lustre-MDT0001 complete [ 4844.236303] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 18:26:22 (1777501582) [ 4846.769678] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4846.903312] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4846.918085] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4847.015970] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4847.020060] Lustre: Skipped 5 previous similar messages [ 4848.568372] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4851.924513] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4851.977428] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4851.989601] Lustre: 212422:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4853.577721] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4855.708500] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4857.072470] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4859.638726] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4859.723874] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4862.095986] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4864.220746] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4878.937567] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4878.970715] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 4881.203591] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4883.226791] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4885.947513] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 2 sec [ 4887.500954] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4887.577716] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4888.945613] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4889.030462] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4890.417292] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff99f106591000.ost_server_uuid 50 [ 4891.025172] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff99f106591000.ost_server_uuid in FULL state after 0 sec [ 4891.426503] Lustre: Permanently deactivating lustre-OST0001 [ 4891.433322] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 4892.356223] Lustre: Permanently reactivating lustre-OST0001 [ 4897.256764] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4897.263220] Lustre: Skipped 8 previous similar messages [ 4897.266514] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4897.270090] LustreError: Skipped 3 previous similar messages [ 4899.581727] Lustre: server umount lustre-OST0001 complete [ 4915.168131] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4915.235339] Lustre: server umount lustre-OST0000 complete [ 4931.040100] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4931.114538] Lustre: server umount lustre-MDT0000 complete [ 4932.600590] LustreError: 214055:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777501671 with bad export cookie 14073438862110009934 [ 4932.602325] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4932.607064] LustreError: 214055:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4932.741507] Lustre: server umount lustre-MDT0001 complete [ 4941.375401] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 18:27:59 (1777501679) [ 4944.171334] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4944.281266] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4944.296319] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4944.301493] Lustre: Skipped 5 previous similar messages [ 4945.936485] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4949.622347] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4949.676391] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4949.689990] Lustre: 218544:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4951.355777] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4953.596215] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4955.080541] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4957.882037] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4957.986996] Lustre: Permanently deactivating lustre-OST0000 [ 4957.996512] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 4960.184531] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4961.207037] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4962.216245] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4964.673213] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4966.796485] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4968.149253] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 4968.796926] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4969.461071] Lustre: Permanently reactivating lustre-OST0000 [ 4975.610934] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 4975.614472] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4975.619433] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4975.623212] Lustre: Skipped 1 previous similar message [ 4975.635824] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 4975.643880] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 4987.206828] Lustre: server umount lustre-OST0001 complete [ 4994.789207] Lustre: server umount lustre-OST0000 complete [ 5004.532349] Lustre: server umount lustre-MDT0000 complete [ 5005.844238] LustreError: 218539:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777501744 with bad export cookie 14073438862110011572 [ 5005.847374] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5005.849407] LustreError: 218539:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5005.964560] Lustre: server umount lustre-MDT0001 complete [ 5009.890784] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 5011.078420] Key type lgssc unregistered [ 5011.207750] LNet: 222947:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5011.211753] LNetError: 222947:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5011.225430] LNet: Removed LNI 192.168.204.106@tcp [ 5011.550456] Key type .llcrypt unregistered [ 5011.552167] Key type ._llcrypt unregistered [ 5021.280457] Key type ._llcrypt registered [ 5021.282223] Key type .llcrypt registered [ 5021.328886] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 5021.781330] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5021.812682] alg: No test for adler32 (adler32-zlib) [ 5022.690104] Lustre: Lustre: Build Version: 2.17.52_93_g6cdb58a [ 5022.796541] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 5024.392203] Key type lgssc registered [ 5024.854839] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5028.975499] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5028.984724] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5030.210362] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5031.750763] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 5035.383571] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5035.459281] Lustre: Found index 1 for lustre-MDT0001, updating log [ 5035.492393] Lustre: 224362:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5035.497753] Lustre: Permanently deactivating lustre-MDT0001 [ 5035.505788] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 5035.652039] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5036.467680] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 5036.471463] LustreError: 224405:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 5036.476144] LustreError: 224405:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 5036.481522] LustreError: 224405:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 6, retries 1, failed: rc = -108 [ 5037.308182] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 5039.557857] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5040.942508] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5043.672351] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5043.826283] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5045.861297] LustreError: 225988:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5045.866797] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 5045.870670] LustreError: 225988:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 5046.195862] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 5048.352164] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5051.328448] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5051.363906] LustreError: 225988:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5051.432544] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5053.772616] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 5055.528895] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 5055.530208] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 5055.530605] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 5056.122418] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5057.857214] Lustre: Permanently reactivating lustre-MDT0001 [ 5057.868969] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 5057.872650] Lustre: Skipped 2 previous similar messages [ 5064.090923] Lustre: lustre-MDT0001: Client 63f37334-7257-4d37-9d20-ec4857ebc383 (at 192.168.204.6@tcp) reconnecting [ 5065.615790] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5066.344268] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5068.420538] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 5068.536543] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 5070.145324] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 5070.256506] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 5071.313353] Lustre: Permanently deactivating lustre-MDT0001 [ 5071.322895] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 5071.325447] Lustre: Skipped 2 previous similar messages [ 5081.105487] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 5086.180649] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5086.182255] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5086.189934] Lustre: Skipped 3 previous similar messages [ 5086.193266] Lustre: Skipped 2 previous similar messages [ 5087.972774] Lustre: server umount lustre-MDT0000 complete [ 5089.480359] LustreError: 224356:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777501828 with bad export cookie 834738821083158011 [ 5089.483651] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5089.486302] LustreError: 224356:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5089.610647] Lustre: server umount lustre-MDT0001 complete [ 5091.144537] Lustre: server umount lustre-OST0000 complete [ 5095.449410] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 18:30:33 (1777501833) [ 5099.307328] Lustre: server umount lustre-OST0001 complete [ 5101.577509] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_hostid [ 5105.322250] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 5120.621021] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5120.735041] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5120.740142] Lustre: Skipped 2 previous similar messages [ 5120.751315] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5120.794678] Lustre: lustre-MDT0000: new disk, initializing [ 5120.826666] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5120.834736] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5122.305295] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 5126.930296] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5126.977849] Lustre: 232053:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5127.005221] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5127.007530] Lustre: Skipped 1 previous similar message [ 5127.048638] Lustre: lustre-MDT0001: new disk, initializing [ 5127.097220] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5127.101579] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5128.598887] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 5132.142613] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5133.539951] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5136.064228] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5136.164638] Lustre: lustre-OST0000: new disk, initializing [ 5136.166773] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5136.194412] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5136.197022] Lustre: Skipped 1 previous similar message [ 5138.178801] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 5141.509967] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5141.514302] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5141.552949] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5141.630900] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5142.541610] LustreError: 232061:0:(mdt_handler.c:3264:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 5147.029830] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5147.100403] Lustre: lustre-OST0001: new disk, initializing [ 5147.102727] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5149.305403] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 5149.772767] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5149.778950] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5149.816346] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5152.640193] LustreError: 232061:0:(mdt_handler.c:3264:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 5152.652992] LustreError: 232060:0:(mdt_handler.c:3264:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 5152.955143] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5153.555909] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5162.744201] LustreError: 232060:0:(mdt_handler.c:3264:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 5164.443855] Lustre: lustre-OST0001: Not available for connect from 192.168.204.6@tcp (stopping) [ 5165.538761] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5165.541454] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5165.542631] LustreError: Skipped 1 previous similar message [ 5165.543396] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5165.547443] Lustre: Skipped 1 previous similar message [ 5169.552380] Lustre: lustre-OST0001: Not available for connect from 192.168.204.6@tcp (stopping) [ 5169.556226] Lustre: Skipped 1 previous similar message [ 5169.606342] Lustre: server umount lustre-OST0001 complete [ 5170.661086] LustreError: 233777:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5170.668641] LustreError: 233777:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 5175.781254] LustreError: 233778:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5175.788148] LustreError: 233778:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 5180.899859] LustreError: 233778:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5180.905499] LustreError: 233778:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 5182.433028] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5182.433221] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5182.435635] LustreError: Skipped 1 previous similar message [ 5182.440948] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5182.443958] Lustre: Skipped 1 previous similar message [ 5191.141123] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5191.142141] LustreError: 234367:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5191.143307] Lustre: Skipped 3 previous similar messages [ 5191.150227] LustreError: 234367:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 5196.256219] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5196.320799] Lustre: server umount lustre-OST0000 complete [ 5198.816838] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5198.820550] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5198.827040] Lustre: Skipped 1 previous similar message [ 5203.939626] Lustre: server umount lustre-MDT0000 complete [ 5205.320244] LustreError: 232045:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777501944 with bad export cookie 834738821083160818 [ 5205.323151] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5205.328481] LustreError: 232045:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5211.543231] Lustre: server umount lustre-MDT0001 complete [ 5216.006436] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 5217.182668] Key type lgssc unregistered [ 5217.329548] LNet: 236695:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5217.334161] LNetError: 236695:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5217.342445] LNet: Removed LNI 192.168.204.106@tcp [ 5217.676078] Key type .llcrypt unregistered [ 5217.678067] Key type ._llcrypt unregistered [ 5233.539368] Key type ._llcrypt registered [ 5233.541170] Key type .llcrypt registered [ 5233.580882] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 5233.880146] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5233.887903] alg: No test for adler32 (adler32-zlib) [ 5234.748607] Lustre: Lustre: Build Version: 2.17.52_93_g6cdb58a [ 5234.840449] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 5236.432114] Key type lgssc registered [ 5236.818683] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5239.553971] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5239.561955] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5240.714451] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5240.732236] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5240.801131] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5242.238247] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 5245.217138] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5245.250531] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5245.261945] Lustre: 238862:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5245.358231] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5246.749842] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 5248.631172] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5249.922910] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5252.200214] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5252.269302] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5252.330164] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5254.260422] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 5256.091716] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5257.222609] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 5303.266904] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5303.267221] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5303.270691] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5303.275290] Lustre: Skipped 1 previous similar message [ 5306.394434] Lustre: server umount lustre-OST0000 complete [ 5307.762593] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 5311.680707] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5311.788067] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5313.582599] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 5315.339653] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5324.067712] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5324.070845] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5324.070928] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5324.075517] Lustre: Skipped 1 previous similar message [ 5349.858131] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5349.858332] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5349.859894] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5349.859898] Lustre: Skipped 1 previous similar message [ 5349.859962] LustreError: Skipped 2 previous similar messages [ 5349.863538] Lustre: Skipped 1 previous similar message [ 5354.517948] Lustre: server umount lustre-OST0000 complete [ 5358.048718] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5358.051499] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5358.056177] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5358.057938] Lustre: Skipped 1 previous similar message [ 5360.096972] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5360.097659] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5361.813764] Lustre: server umount lustre-MDT0000 complete [ 5363.014103] LustreError: 242774:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777502101 with bad export cookie 4695139324010987280 [ 5363.016880] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5363.017547] LustreError: 242774:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5363.134720] Lustre: server umount lustre-MDT0001 complete [ 5366.569773] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 5367.601082] Key type lgssc unregistered [ 5367.724499] LNet: 243808:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5367.728190] LNetError: 243808:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5367.737334] LNet: Removed LNI 192.168.204.106@tcp [ 5368.026217] Key type .llcrypt unregistered [ 5368.027657] Key type ._llcrypt unregistered [ 5376.776615] Key type ._llcrypt registered [ 5376.778155] Key type .llcrypt registered [ 5376.826332] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 5377.343606] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5377.365432] alg: No test for adler32 (adler32-zlib) [ 5378.248974] Lustre: Lustre: Build Version: 2.17.52_93_g6cdb58a [ 5378.336349] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 5379.920289] Key type lgssc registered [ 5380.357703] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5383.321280] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5383.329298] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5384.508088] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5385.911704] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 5388.899147] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5389.022177] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5390.311267] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 5392.051926] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5393.208206] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5395.451587] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5395.593172] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5397.471924] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 5399.316774] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5410.987296] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 5424.608853] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5424.612104] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5424.615862] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5426.657874] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5426.658374] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5426.660038] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5428.172445] Lustre: server umount lustre-OST0000 complete [ 5430.240723] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5430.242924] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5430.246713] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5430.249020] Lustre: Skipped 1 previous similar message [ 5435.482447] Lustre: server umount lustre-MDT0000 complete [ 5436.731229] LustreError: 245166:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777502175 with bad export cookie 17944110922001610292 [ 5436.734030] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5436.736812] LustreError: 245166:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5436.845785] Lustre: server umount lustre-MDT0001 complete [ 5440.546829] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 5441.577494] Key type lgssc unregistered [ 5441.713459] LNet: 249154:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5441.716523] LNetError: 249154:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5441.726418] LNet: Removed LNI 192.168.204.106@tcp [ 5442.036771] Key type .llcrypt unregistered [ 5442.038466] Key type ._llcrypt unregistered [ 5449.147530] Key type ._llcrypt registered [ 5449.148878] Key type .llcrypt registered [ 5449.187877] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 5449.598805] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5449.608326] alg: No test for adler32 (adler32-zlib) [ 5450.461085] Lustre: Lustre: Build Version: 2.17.52_93_g6cdb58a [ 5450.539712] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 5452.120158] Key type lgssc registered [ 5452.491955] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5454.986487] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5454.992348] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5456.139597] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5457.387760] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 5460.199660] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5460.331990] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5461.637646] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 5463.439290] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5464.615602] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5466.885102] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5466.994203] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5468.815594] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 5470.555717] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5480.294763] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5485.537790] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5485.538474] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5485.539785] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5485.546271] Lustre: Skipped 1 previous similar message [ 5486.351488] Lustre: server umount lustre-OST0000 complete [ 5489.677186] Lustre: server umount lustre-MDT0000 complete [ 5490.657135] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5490.657657] LustreError: 252484:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5490.661390] Lustre: Skipped 1 previous similar message [ 5490.667162] LustreError: 252484:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 5490.867809] LustreError: 250316:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777502229 with bad export cookie 328954258219853314 [ 5490.869942] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5490.874170] LustreError: 250316:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5490.990870] Lustre: server umount lustre-MDT0001 complete [ 5494.761788] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 5495.840074] Key type lgssc unregistered [ 5495.977653] LNet: 253671:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5495.981047] LNetError: 253671:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5495.988634] LNet: Removed LNI 192.168.204.106@tcp [ 5496.282237] Key type .llcrypt unregistered [ 5496.283835] Key type ._llcrypt unregistered [ 5505.194964] Key type ._llcrypt registered [ 5505.197569] Key type .llcrypt registered [ 5505.245124] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 5505.585022] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5505.637719] alg: No test for adler32 (adler32-zlib) [ 5506.485763] Lustre: Lustre: Build Version: 2.17.52_93_g6cdb58a [ 5506.571159] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 5508.152173] Key type lgssc registered [ 5508.532543] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5511.112241] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5511.118606] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5512.283224] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5513.516167] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 5516.313537] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5516.447241] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5517.763879] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 5519.463424] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5520.661703] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5522.859698] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5522.915480] LustreError: 256638:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5522.920821] LustreError: 256638:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 5522.976846] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5524.933278] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 5526.714954] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5527.079306] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 5540.834266] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 5540.837112] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5540.841592] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5542.370296] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5542.371908] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5542.373113] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5546.389721] Lustre: server umount lustre-OST0000 complete [ 5552.610807] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5552.610827] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5552.611618] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5552.611623] Lustre: Skipped 1 previous similar message [ 5552.620230] Lustre: Skipped 1 previous similar message [ 5553.818410] Lustre: server umount lustre-MDT0000 complete [ 5555.127265] LustreError: 255030:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777502293 with bad export cookie 12399968036986797151 [ 5555.131318] LustreError: 255030:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5555.131719] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5555.232590] Lustre: server umount lustre-MDT0001 complete [ 5558.715270] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 5559.747243] Key type lgssc unregistered [ 5559.870353] LNet: 258991:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5559.872869] LNetError: 258991:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5559.879305] LNet: Removed LNI 192.168.204.106@tcp [ 5560.164561] Key type .llcrypt unregistered [ 5560.165945] Key type ._llcrypt unregistered [ 5567.389161] Key type ._llcrypt registered [ 5567.390168] Key type .llcrypt registered [ 5567.429224] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 5567.752967] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5567.809509] alg: No test for adler32 (adler32-zlib) [ 5568.666816] Lustre: Lustre: Build Version: 2.17.52_93_g6cdb58a [ 5568.752252] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 5570.328150] Key type lgssc registered [ 5570.717721] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5573.336250] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5573.343507] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5574.513370] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5575.742737] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 5578.533917] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5578.652623] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5579.914922] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 5581.625656] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5582.747653] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5585.002823] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5585.127149] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5587.022240] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 5588.805470] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5599.462337] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 5608.928914] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5608.932303] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5608.937893] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5609.953837] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5609.954055] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5609.955230] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5609.964230] Lustre: Skipped 1 previous similar message [ 5614.101210] Lustre: server umount lustre-OST0000 complete [ 5619.680916] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5619.684113] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5619.690174] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5621.526214] Lustre: server umount lustre-MDT0000 complete [ 5622.776152] LustreError: 260154:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777502361 with bad export cookie 8396955190954940633 [ 5622.778749] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5622.780447] LustreError: 260154:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5622.902770] Lustre: server umount lustre-MDT0001 complete [ 5626.933079] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 5628.094662] Key type lgssc unregistered [ 5628.230389] LNet: 263552:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5628.233287] LNetError: 263552:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5628.243301] LNet: Removed LNI 192.168.204.106@tcp [ 5628.566540] Key type .llcrypt unregistered [ 5628.567793] Key type ._llcrypt unregistered [ 5632.527973] Key type ._llcrypt registered [ 5632.529017] Key type .llcrypt registered [ 5632.568519] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 5638.572108] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_hostid [ 5644.751461] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 5645.140835] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5645.146748] alg: No test for adler32 (adler32-zlib) [ 5645.993124] Lustre: Lustre: Build Version: 2.17.52_93_g6cdb58a [ 5646.075914] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 5647.656143] Key type lgssc registered [ 5647.991291] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5659.014048] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5659.020629] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5660.106825] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5660.118730] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5660.153587] Lustre: lustre-MDT0000: new disk, initializing [ 5660.175364] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5660.182385] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5661.432420] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 5665.638812] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5665.682975] Lustre: 267024:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5665.708288] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5665.711302] Lustre: Skipped 1 previous similar message [ 5665.750595] Lustre: lustre-MDT0001: new disk, initializing [ 5665.783885] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5665.796648] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5665.800817] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5667.192527] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 5670.382492] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5671.667665] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5673.987737] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5674.089213] Lustre: lustre-OST0000: new disk, initializing [ 5674.091239] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5674.123806] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5675.646143] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5675.650548] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5675.694189] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5676.206369] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 5679.446135] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5680.964328] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5681.055156] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5682.482447] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5682.564285] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5686.244959] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5686.248089] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5686.253803] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5686.256265] Lustre: Skipped 1 previous similar message [ 5689.369079] Lustre: server umount lustre-OST0000 complete [ 5691.363554] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5691.363701] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5691.364451] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5691.367804] LustreError: Skipped 1 previous similar message [ 5691.373421] Lustre: Skipped 2 previous similar messages [ 5696.481190] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5696.483905] Lustre: Skipped 2 previous similar messages [ 5696.861153] Lustre: server umount lustre-MDT0000 complete [ 5698.226366] LustreError: 267803:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777502437 with bad export cookie 1407361919242785857 [ 5698.229749] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5698.231207] LustreError: 267803:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5698.346331] Lustre: server umount lustre-MDT0001 complete [ 5701.429697] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 18:40:39 (1777502439) [ 5703.674034] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5703.864301] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5705.206237] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 5708.146802] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5708.270568] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5709.631726] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 5711.478895] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5712.703588] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5715.061788] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5717.205677] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 5719.072686] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5733.859457] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5733.864623] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5733.864930] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5733.868822] Lustre: Skipped 1 previous similar message [ 5738.638849] Lustre: server umount lustre-OST0000 complete [ 5743.074071] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 5743.076839] LustreError: Skipped 1 previous similar message [ 5743.078484] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5743.083968] Lustre: Skipped 1 previous similar message [ 5743.085856] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5753.313208] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5753.315843] Lustre: Skipped 5 previous similar messages [ 5754.336091] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5754.393092] Lustre: server umount lustre-MDT0000 complete [ 5755.635238] LustreError: 271052:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777502494 with bad export cookie 1407361919242786725 [ 5755.637989] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5755.639397] LustreError: 271052:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5755.746587] Lustre: server umount lustre-MDT0001 complete [ 5759.280349] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 5760.273498] Key type lgssc unregistered [ 5760.405386] LNet: 274223:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5760.408493] LNetError: 274223:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5760.418345] LNet: Removed LNI 192.168.204.106@tcp [ 5760.695499] Key type .llcrypt unregistered [ 5760.696629] Key type ._llcrypt unregistered [ 5769.426671] Key type ._llcrypt registered [ 5769.427701] Key type .llcrypt registered [ 5769.462734] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_hostid [ 5775.352701] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 5775.638753] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5775.648384] alg: No test for adler32 (adler32-zlib) [ 5776.492876] Lustre: Lustre: Build Version: 2.17.52_93_g6cdb58a [ 5776.583598] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 5778.168164] Key type lgssc registered [ 5778.594255] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5793.913126] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5793.918652] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5795.009512] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5795.025453] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5795.067519] Lustre: lustre-MDT0000: new disk, initializing [ 5795.099122] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5795.108384] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5796.504885] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 5800.939342] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5800.989254] Lustre: 277986:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5801.021874] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5801.024426] Lustre: Skipped 1 previous similar message [ 5801.064622] Lustre: lustre-MDT0001: new disk, initializing [ 5801.105143] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5801.118617] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5801.122964] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5802.522288] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 5805.817949] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5807.047661] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5809.416861] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5809.538631] Lustre: lustre-OST2710: new disk, initializing [ 5809.540706] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 5809.578065] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 5810.487822] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 5810.491749] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 5810.530538] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 5811.523753] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 5814.750650] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 5817.195634] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5817.255362] Lustre: lustre-OST03e8: new disk, initializing [ 5817.257203] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 5817.290728] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 5818.687824] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 5818.692363] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 5818.721226] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 5819.170618] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 5820.950150] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 5823.531752] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 5823.601294] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5824.846808] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 5824.911097] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5826.144225] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 5826.219736] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5827.589232] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 5827.661139] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5829.093106] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5829.095619] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5829.096433] Lustre: Skipped 3 previous similar messages [ 5829.098890] Lustre: Skipped 3 previous similar messages [ 5834.211248] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5834.213107] Lustre: Skipped 2 previous similar messages [ 5834.840493] Lustre: server umount lustre-MDT0000 complete [ 5836.049996] LustreError: 278764:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777502574 with bad export cookie 6504367479182617195 [ 5836.052568] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5836.054455] LustreError: 278764:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5836.163571] Lustre: server umount lustre-MDT0001 complete [ 5847.556610] Lustre: server umount lustre-OST2710 complete [ 5859.098131] Lustre: server umount lustre-OST03e8 complete [ 5864.950787] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_hostid [ 5868.189373] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 5880.086024] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5880.177755] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5880.191571] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5880.226304] Lustre: lustre-MDT0000: new disk, initializing [ 5880.250045] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5880.256472] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5881.558202] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 5885.633319] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5885.672861] Lustre: 285700:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5885.761937] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5886.941942] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 5889.875190] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5891.018460] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5893.148098] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5893.237352] Lustre: lustre-OST0000: new disk, initializing [ 5893.239384] Lustre: Skipped 1 previous similar message [ 5893.241626] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5893.244249] Lustre: Skipped 2 previous similar messages [ 5893.269269] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5893.271752] Lustre: Skipped 1 previous similar message [ 5894.968091] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 5897.956947] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5899.342615] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5900.789962] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5900.792639] Lustre: Skipped 1 previous similar message [ 5900.794444] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5900.816603] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5901.441521] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 5902.631285] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5902.700769] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5905.891634] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5905.893158] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5905.895470] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5905.896896] LustreError: Skipped 1 previous similar message [ 5905.899909] Lustre: Skipped 2 previous similar messages [ 5909.458221] Lustre: server umount lustre-OST0000 complete [ 5911.009271] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5911.009863] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5911.012416] Lustre: Skipped 2 previous similar messages [ 5911.015438] Lustre: Skipped 1 previous similar message [ 5916.128942] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5916.130951] Lustre: Skipped 1 previous similar message [ 5916.820871] Lustre: server umount lustre-MDT0000 complete [ 5917.984340] LustreError: 286479:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777502656 with bad export cookie 6504367479182618861 [ 5917.987523] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5917.988762] LustreError: 286479:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5918.093819] Lustre: server umount lustre-MDT0001 complete [ 5920.684315] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 18:44:18 (1777502658) [ 5921.182297] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 5921.708485] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 18:44:19 (1777502659) [ 5922.613530] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 5931.065626] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5931.146405] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5931.153556] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5931.207917] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5932.393366] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 5935.088967] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5935.121219] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5935.130755] Lustre: 291138:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5936.430644] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 5938.136237] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5939.317544] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5954.530226] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5954.587871] Lustre: Denying initial registration attempt from nid 192.168.204.106@tcp, specified as failover [ 5954.590791] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 5954.593584] LustreError: 292752:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -99 [ 5954.595540] LustreError: 292752:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 5954.597278] LustreError: 292752:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 5954.618688] Lustre: server umount lustre-OST0000 complete [ 5954.619916] LustreError: 292752:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 5961.185513] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5961.185642] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5961.186314] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5961.193157] Lustre: Skipped 1 previous similar message [ 5962.645590] Lustre: server umount lustre-MDT0000 complete [ 5963.896564] LustreError: 291131:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777502702 with bad export cookie 6504367479182619729 [ 5963.900215] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5963.901562] LustreError: 291131:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5964.017496] Lustre: server umount lustre-MDT0001 complete [ 5967.453157] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 5968.515548] Key type lgssc unregistered [ 5968.649347] LNet: 293854:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5968.652301] LNetError: 293854:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5968.659344] LNet: Removed LNI 192.168.204.106@tcp [ 5968.950830] Key type .llcrypt unregistered [ 5968.952159] Key type ._llcrypt unregistered [ 5972.442903] Key type ._llcrypt registered [ 5972.444748] Key type .llcrypt registered [ 5972.485939] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 5972.878952] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5972.935295] alg: No test for adler32 (adler32-zlib) [ 5973.802132] Lustre: Lustre: Build Version: 2.17.52_93_g6cdb58a [ 5973.893986] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 5975.480182] Key type lgssc registered [ 5975.881380] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6010.864657] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 6013.733107] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6013.738903] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6014.825937] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6014.834677] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6014.893550] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6016.169304] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 6019.036984] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6019.074408] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6019.086435] Lustre: 296569:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6019.186271] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6020.553686] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 6022.486839] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6023.759734] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6039.074742] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6039.140103] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6039.197091] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6041.031568] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 6042.853707] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6043.624476] Lustre: server umount lustre-OST0000 complete [ 6045.152785] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6045.155859] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6045.160817] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6046.190960] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6046.191376] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6051.028170] Lustre: server umount lustre-MDT0000 complete [ 6052.288242] LustreError: 298205:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777502791 with bad export cookie 9874576602916332328 [ 6052.291460] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6052.292614] LustreError: 298205:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6052.407070] Lustre: server umount lustre-MDT0001 complete [ 6055.814042] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 6056.870399] Key type lgssc unregistered [ 6057.000325] LNet: 299774:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6057.004107] LNetError: 299774:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6057.012294] LNet: Removed LNI 192.168.204.106@tcp [ 6057.308313] Key type .llcrypt unregistered [ 6057.310204] Key type ._llcrypt unregistered [ 6066.438446] Key type ._llcrypt registered [ 6066.440039] Key type .llcrypt registered [ 6066.476410] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 6066.768258] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6066.776598] alg: No test for adler32 (adler32-zlib) [ 6067.627037] Lustre: Lustre: Build Version: 2.17.52_93_g6cdb58a [ 6067.708030] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 6069.288121] Key type lgssc registered [ 6069.623375] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6072.079124] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6072.086290] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6073.246537] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6074.472537] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 6077.281883] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6077.404387] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6078.698700] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 6080.474553] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6081.702132] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6086.051255] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6086.163278] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6088.098677] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 6090.013808] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6103.008829] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6103.011893] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6103.017775] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6103.522336] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6103.523126] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6103.527386] Lustre: Skipped 1 previous similar message [ 6103.531664] Lustre: Skipped 1 previous similar message [ 6105.494543] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.6@tcp (stopping) [ 6108.641495] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6108.643824] Lustre: Skipped 2 previous similar messages [ 6108.949258] Lustre: server umount lustre-MDT0000 complete [ 6110.271985] LustreError: 301131:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777502849 with bad export cookie 5673946650900701133 [ 6110.275284] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6110.276401] LustreError: 301131:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6110.607304] LustreError: 301146:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6110.614276] LustreError: 301146:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 6113.760791] LustreError: 301145:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6113.761028] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6113.771331] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6113.773116] Lustre: Skipped 1 previous similar message [ 6115.727446] LustreError: 301145:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6116.456491] Lustre: server umount lustre-MDT0001 complete [ 6118.598783] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 6121.355756] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6135.264701] LustreError: 300460:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000001a4c5d46 x1863846782329856/t0(0) o250->MGC192.168.204.106@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6135.400973] LustreError: 302753:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6135.407798] LustreError: 302753:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 6135.425944] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6135.447163] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:161) [ 6136.781777] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 6139.691197] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6139.812224] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6140.837315] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6141.098583] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 6142.915063] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6145.956805] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6147.549759] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 4 sec [ 6148.756510] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6149.309968] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6151.141906] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6151.142918] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6151.143256] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6151.143263] Lustre: Skipped 2 previous similar messages [ 6151.146540] Lustre: Skipped 1 previous similar message [ 6156.689314] Lustre: server umount lustre-OST0000 complete [ 6160.353048] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6160.356853] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6164.116706] Lustre: server umount lustre-MDT0000 complete [ 6165.392243] LustreError: 306113:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777502904 with bad export cookie 5673946650900708287 [ 6165.395779] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6165.396100] LustreError: 306113:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6165.520892] Lustre: server umount lustre-MDT0001 complete [ 6169.125381] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 6170.158703] Key type lgssc unregistered [ 6170.286297] LNet: 306846:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6170.289500] LNetError: 306846:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6170.297306] LNet: Removed LNI 192.168.204.106@tcp [ 6170.578855] Key type .llcrypt unregistered [ 6170.579963] Key type ._llcrypt unregistered [ 6179.338643] Key type ._llcrypt registered [ 6179.339912] Key type .llcrypt registered [ 6179.382202] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 6179.840378] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6179.855166] alg: No test for adler32 (adler32-zlib) [ 6180.703308] Lustre: Lustre: Build Version: 2.17.52_93_g6cdb58a [ 6180.781974] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 6182.368161] Key type lgssc registered [ 6182.721956] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6185.337176] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6185.343791] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6186.518479] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6187.892072] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 6190.724820] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6190.857614] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6192.175746] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 6193.948146] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6195.161778] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6196.156546] LustreError: 308251:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 6196.160295] LustreError: 308251:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 6196.162956] LustreError: 308251:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 9, retries 0, failed: rc = -5 [ 6196.193512] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6196.193613] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6196.194216] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6196.199053] Lustre: Skipped 1 previous similar message [ 6200.289181] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6200.291639] Lustre: Skipped 2 previous similar messages [ 6202.320825] Lustre: server umount lustre-MDT0000 complete [ 6203.554438] LustreError: 308202:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777502942 with bad export cookie 17224794352273788874 [ 6203.559176] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6203.560341] LustreError: 308202:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6203.674422] Lustre: server umount lustre-MDT0001 complete [ 6208.491695] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6208.585380] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6208.596647] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6208.669029] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6209.950577] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 6212.765270] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6212.797071] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6212.810679] Lustre: 310581:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6212.903403] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6214.123835] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 6215.826594] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6217.063754] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6221.670074] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6221.739645] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 6221.743892] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 6221.747091] LustreError: 312235:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -2 [ 6221.749698] LustreError: 312235:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 6221.752599] LustreError: 312235:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 6221.778658] Lustre: server umount lustre-OST0000 complete [ 6221.780924] LustreError: 312235:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 6226.658937] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6226.721794] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6226.782659] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6228.669639] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 6230.386733] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6232.963885] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6232.994717] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6233.021253] Lustre: lustre-OST0001: new disk, initializing [ 6233.023479] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6233.676630] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6233.682225] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6233.704278] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:193) [ 6233.732784] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6234.958138] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 6236.806971] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6239.207346] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6239.208346] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6239.211115] LustreError: Skipped 1 previous similar message [ 6239.216632] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6239.220151] Lustre: Skipped 1 previous similar message [ 6239.872724] Lustre: server umount lustre-OST0001 complete [ 6243.809182] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6243.812311] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6243.816119] Lustre: Skipped 1 previous similar message [ 6243.817931] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6244.323182] LustreError: 312701:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6244.329494] LustreError: 312701:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 6247.246678] Lustre: server umount lustre-OST0000 complete [ 6248.929106] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6248.932275] LustreError: Skipped 1 previous similar message [ 6248.934071] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6248.939152] Lustre: Skipped 1 previous similar message [ 6248.941502] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6248.943956] Lustre: Skipped 2 previous similar messages [ 6254.814102] Lustre: server umount lustre-MDT0000 complete [ 6256.029149] LustreError: 310571:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777502994 with bad export cookie 17224794352273789399 [ 6256.031468] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6256.033087] LustreError: 310571:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6256.147309] Lustre: server umount lustre-MDT0001 complete [ 6259.546378] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 6260.566654] Key type lgssc unregistered [ 6260.692381] LNet: 315480:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6260.695322] LNetError: 315480:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6260.707317] LNet: Removed LNI 192.168.204.106@tcp [ 6260.995669] Key type .llcrypt unregistered [ 6260.997169] Key type ._llcrypt unregistered [ 6291.750807] Key type ._llcrypt registered [ 6291.752419] Key type .llcrypt registered [ 6291.788717] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_hostid [ 6298.033247] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 6298.409786] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6298.443941] alg: No test for adler32 (adler32-zlib) [ 6299.319039] Lustre: Lustre: Build Version: 2.17.52_93_g6cdb58a [ 6299.412843] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 6301.000231] Key type lgssc registered [ 6301.435245] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6313.660833] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6313.668419] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6314.759122] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6314.772617] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6314.809372] Lustre: lustre-MDT0000: new disk, initializing [ 6314.838313] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6314.845561] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6316.180530] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 6320.541910] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6320.583819] Lustre: 320355:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6320.608180] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6320.612044] Lustre: Skipped 1 previous similar message [ 6320.649604] Lustre: lustre-MDT0001: new disk, initializing [ 6320.684632] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6320.698205] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6320.703439] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6321.980040] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 6325.089721] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6326.245903] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6328.441659] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6328.540303] Lustre: lustre-OST0000: new disk, initializing [ 6328.542446] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6328.569617] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6330.355957] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 6333.408516] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6334.726803] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6338.038320] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6338.042576] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6338.068470] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6338.863322] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 4 sec [ 6340.176813] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6340.249344] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6343.139205] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6343.139550] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6343.144142] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6343.148523] Lustre: Skipped 1 previous similar message [ 6346.953542] Lustre: server umount lustre-OST0000 complete [ 6348.257071] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6348.257613] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6348.260411] Lustre: Skipped 2 previous similar messages [ 6353.377123] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6353.379628] Lustre: Skipped 2 previous similar messages [ 6354.271259] Lustre: server umount lustre-MDT0000 complete [ 6355.524100] LustreError: 320349:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777503094 with bad export cookie 4998099339656998874 [ 6355.526696] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6355.528199] LustreError: 320349:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6355.641721] Lustre: server umount lustre-MDT0001 complete [ 6358.282447] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 18:51:36 (1777503096) [ 6360.732045] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 18:51:38 (1777503098) [ 6363.593984] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6363.788977] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6365.134866] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 6368.046781] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6368.170107] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6369.540367] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 6371.339925] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6372.483104] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6374.834672] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6376.832410] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 6378.549033] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6392.480699] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 6393.032611] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 6393.565231] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 6394.132540] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 6394.832462] Lustre: Failing over lustre-MDT0000 [ 6394.978632] Lustre: server umount lustre-MDT0000 complete [ 6398.434899] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6398.435738] LustreError: 325442:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6398.439565] Lustre: Skipped 2 previous similar messages [ 6398.447112] LustreError: 325442:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 4 previous similar messages [ 6399.382326] LustreError: 326077:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6403.554291] LustreError: 324732:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6403.561736] LustreError: 324732:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 6407.907728] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6407.952085] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6408.050312] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6408.052976] Lustre: Skipped 1 previous similar message [ 6408.076468] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6409.433714] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 6409.616513] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6413.288797] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6413.297054] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 6413.314022] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6414.459327] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6414.993414] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6416.817332] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 6417.494616] Lustre: Failing over lustre-MDT0000 [ 6417.641643] Lustre: server umount lustre-MDT0000 complete [ 6418.401272] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6418.401872] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6418.402908] LustreError: 324733:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6418.402916] LustreError: 324733:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 6418.403901] LustreError: Skipped 1 previous similar message [ 6418.416014] Lustre: Skipped 2 previous similar messages [ 6423.521683] LustreError: 325442:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6423.526698] LustreError: 325442:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 5 previous similar messages [ 6430.644839] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6430.689560] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6430.781429] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6430.800446] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6432.119424] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 6433.692430] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6435.818760] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6435.822211] Lustre: Skipped 2 previous similar messages [ 6435.823351] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 6435.843219] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 6437.033236] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6437.569664] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6440.929988] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6440.930599] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6440.931399] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6440.934318] Lustre: Skipped 2 previous similar messages [ 6445.905735] Lustre: server umount lustre-MDT0000 complete [ 6446.048926] LustreError: 324732:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6446.054329] LustreError: 324732:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 7 previous similar messages [ 6447.031242] LustreError: 324714:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777503185 with bad export cookie 4998099339657001527 [ 6447.033163] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6447.036094] LustreError: 324714:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6447.138698] Lustre: server umount lustre-MDT0001 complete [ 6458.585820] Lustre: server umount lustre-OST0000 complete [ 6463.440721] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6463.584589] LustreError: 330514:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6463.604627] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6464.853760] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 6467.515572] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6468.915945] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 6470.651296] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6471.746755] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6473.950371] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6475.799618] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 6477.490363] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6488.422232] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 6498.785523] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6498.786252] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6498.787666] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6498.787671] Lustre: Skipped 2 previous similar messages [ 6498.789698] LustreError: Skipped 1 previous similar message [ 6498.793131] Lustre: Skipped 1 previous similar message [ 6500.434857] Lustre: server umount lustre-OST0000 complete [ 6503.392860] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6507.859675] Lustre: server umount lustre-MDT0000 complete [ 6509.024963] LustreError: 332673:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6509.030085] LustreError: 332673:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 6509.103927] LustreError: 330495:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777503247 with bad export cookie 4998099339657002269 [ 6509.106606] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6509.107314] LustreError: 330495:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6509.214637] Lustre: server umount lustre-MDT0001 complete [ 6512.934649] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 6513.929592] Key type lgssc unregistered [ 6514.062288] LNet: 333708:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6514.064732] LNetError: 333708:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6514.074266] LNet: Removed LNI 192.168.204.106@tcp [ 6514.372743] Key type .llcrypt unregistered [ 6514.373886] Key type ._llcrypt unregistered [ 6524.238732] Key type ._llcrypt registered [ 6524.239857] Key type .llcrypt registered [ 6524.278531] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 6524.705257] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6524.728273] alg: No test for adler32 (adler32-zlib) [ 6525.577685] Lustre: Lustre: Build Version: 2.17.52_93_g6cdb58a [ 6525.655540] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 6527.232122] Key type lgssc registered [ 6527.586165] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6530.100703] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6530.107510] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6531.264080] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6532.485559] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 6535.356228] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6535.487731] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6536.841272] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 6538.633756] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6539.966962] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6542.339408] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6542.452834] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6544.368702] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 6546.115826] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6556.773373] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 6566.369460] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6566.371967] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6566.376234] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6567.393584] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6567.395247] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6567.397181] Lustre: Skipped 1 previous similar message [ 6567.404162] Lustre: Skipped 1 previous similar message [ 6571.794909] Lustre: server umount lustre-MDT0000 complete [ 6572.512973] LustreError: 335184:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6572.517912] LustreError: 335184:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 4 previous similar messages [ 6572.919127] LustreError: 335166:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777503311 with bad export cookie 9516773091563101318 [ 6572.922208] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6572.924262] LustreError: 335166:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6573.030184] Lustre: server umount lustre-MDT0001 complete [ 6584.469625] Lustre: server umount lustre-OST0000 complete [ 6588.606069] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6588.739789] LustreError: 338575:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6588.760101] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6589.941754] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 6592.677749] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6592.779243] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6593.963147] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 6595.572943] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6596.690470] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6598.826537] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6599.976738] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 6600.728359] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 6602.434161] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6613.474672] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6613.474912] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6613.475599] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6613.483008] Lustre: Skipped 2 previous similar messages [ 6615.185488] Lustre: server umount lustre-MDT0000 complete [ 6616.273208] LustreError: 338556:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777503355 with bad export cookie 9516773091563102417 [ 6616.276314] LustreError: 338556:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6616.277325] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6616.375297] Lustre: server umount lustre-MDT0001 complete [ 6627.798341] Lustre: server umount lustre-OST0000 complete [ 6632.923036] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6633.094085] LustreError: 342142:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6633.121741] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6633.124116] Lustre: Skipped 1 previous similar message [ 6634.411360] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 6637.245991] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6638.749152] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 6640.545523] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6641.712516] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6644.043601] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6646.027384] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 6647.796116] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6657.511219] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 6662.626584] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6662.627842] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6662.632550] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6662.633910] LustreError: Skipped 1 previous similar message [ 6662.635704] Lustre: Skipped 3 previous similar messages [ 6665.815105] Lustre: server umount lustre-OST0000 complete [ 6667.745137] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6667.749171] Lustre: Skipped 1 previous similar message [ 6667.751209] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6667.752974] Lustre: Skipped 1 previous similar message [ 6673.177570] Lustre: server umount lustre-MDT0000 complete [ 6674.406401] LustreError: 344603:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777503413 with bad export cookie 9516773091563103425 [ 6674.410234] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6674.410943] LustreError: 344603:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6674.536394] Lustre: server umount lustre-MDT0001 complete [ 6678.205595] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 6679.223835] Key type lgssc unregistered [ 6679.359421] LNet: 345337:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6679.362444] LNetError: 345337:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6679.369362] LNet: Removed LNI 192.168.204.106@tcp [ 6679.684087] Key type .llcrypt unregistered [ 6679.685771] Key type ._llcrypt unregistered [ 6689.205747] Key type ._llcrypt registered [ 6689.206879] Key type .llcrypt registered [ 6689.248456] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 6689.566995] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6689.574646] alg: No test for adler32 (adler32-zlib) [ 6690.425775] Lustre: Lustre: Build Version: 2.17.52_93_g6cdb58a [ 6690.516874] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 6692.096158] Key type lgssc registered [ 6692.464633] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6695.132277] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6695.138143] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6695.140827] LustreError: 346737:0:(osd_handler.c:8530:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 6695.149152] LustreError: 346737:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 6695.151496] LustreError: 346737:0:(obd_mount.c:250:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 6695.153668] LustreError: 346737:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 6695.156521] LustreError: 346737:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 6697.585442] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6697.587626] LustreError: 347133:0:(osd_handler.c:8530:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 6697.594955] LustreError: 347133:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [ 6697.596905] LustreError: 347133:0:(obd_mount.c:250:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 6697.599128] LustreError: 347133:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 6697.601500] LustreError: 347133:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 6704.036119] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 6705.000257] Key type lgssc unregistered [ 6705.113282] LNet: 348117:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6705.115785] LNetError: 348117:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6705.122286] LNet: Removed LNI 192.168.204.106@tcp [ 6705.392406] Key type .llcrypt unregistered [ 6705.393471] Key type ._llcrypt unregistered [ 6710.969515] Key type ._llcrypt registered [ 6710.971051] Key type .llcrypt registered [ 6711.006378] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_hostid [ 6716.715487] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 6717.012397] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6717.050330] alg: No test for adler32 (adler32-zlib) [ 6717.898244] Lustre: Lustre: Build Version: 2.17.52_93_g6cdb58a [ 6717.975367] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 6719.552102] Key type lgssc registered [ 6719.848311] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6730.676230] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6730.682026] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6731.770804] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6731.784316] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6731.822389] Lustre: lustre-MDT0000: new disk, initializing [ 6731.846607] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6731.852986] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6733.102199] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 6737.161107] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6737.203827] Lustre: 351211:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6737.236614] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 6737.237226] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6737.241045] Lustre: Skipped 1 previous similar message [ 6737.279762] Lustre: lustre-MDT0001: new disk, initializing [ 6737.311683] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6737.323493] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6737.326842] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6738.568429] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 6741.663909] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6742.947260] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6745.271216] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6745.371170] Lustre: lustre-OST0000: new disk, initializing [ 6745.373340] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6745.403678] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6746.104156] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6746.108775] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6746.140523] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6747.207182] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 6750.372436] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6751.708174] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6751.792074] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6753.243654] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6753.318520] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6756.321829] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6756.322545] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6756.324788] LustreError: Skipped 1 previous similar message [ 6756.325613] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6756.328088] Lustre: Skipped 1 previous similar message [ 6760.085431] Lustre: server umount lustre-OST0000 complete [ 6761.441417] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6761.442044] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6761.446541] Lustre: Skipped 1 previous similar message [ 6761.448358] Lustre: Skipped 1 previous similar message [ 6766.561307] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6766.563201] Lustre: Skipped 2 previous similar messages [ 6767.515706] Lustre: server umount lustre-MDT0000 complete [ 6768.728556] LustreError: 352004:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777503507 with bad export cookie 9116650378358493930 [ 6768.731927] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6768.734939] LustreError: 352004:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6768.858385] Lustre: server umount lustre-MDT0001 complete [ 6771.717893] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 6772.333833] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 18:58:30 (1777503510) [ 6774.636257] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6774.828324] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6776.152462] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 6778.986829] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6779.113796] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6780.453818] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 6782.197713] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6783.343990] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6785.595500] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6787.622921] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 6789.388922] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6791.696525] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6791.770597] Lustre: lustre-OST0001: new disk, initializing [ 6791.772518] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6791.808450] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6791.810604] Lustre: Skipped 1 previous similar message [ 6793.659438] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 6796.890238] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6798.336580] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6798.340154] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6798.374399] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6803.425160] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6803.425287] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6803.427655] LustreError: Skipped 1 previous similar message [ 6803.431995] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6803.434709] Lustre: Skipped 1 previous similar message [ 6805.054463] Lustre: server umount lustre-OST0001 complete [ 6805.726099] LustreError: 356851:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6805.732046] LustreError: 356851:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 6809.573092] LustreError: 356851:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6809.578298] LustreError: 356851:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 6814.695599] LustreError: 356853:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6814.700764] LustreError: 356853:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 6819.810300] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6819.812792] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6819.813082] LustreError: 356853:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6819.814166] Lustre: Skipped 2 previous similar messages [ 6819.814292] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6819.814297] Lustre: Skipped 2 previous similar messages [ 6819.817364] LustreError: Skipped 1 previous similar message [ 6823.249019] Lustre: server umount lustre-OST0000 complete [ 6824.929457] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6830.678805] Lustre: server umount lustre-MDT0000 complete [ 6831.948090] LustreError: 355240:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777503570 with bad export cookie 9116650378358494784 [ 6831.950935] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6831.952388] LustreError: 355240:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6838.134487] Lustre: server umount lustre-MDT0001 complete [ 6841.812253] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 6842.817438] Key type lgssc unregistered [ 6842.944866] LNet: 359607:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6842.950325] LNetError: 359607:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6842.961503] LNet: Removed LNI 192.168.204.106@tcp [ 6843.401509] Key type .llcrypt unregistered [ 6843.403967] Key type ._llcrypt unregistered [ 6861.762652] Key type ._llcrypt registered [ 6861.763790] Key type .llcrypt registered [ 6861.801850] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 6862.119750] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6862.169628] alg: No test for adler32 (adler32-zlib) [ 6863.027536] Lustre: Lustre: Build Version: 2.17.52_93_g6cdb58a [ 6863.105784] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 6864.688125] Key type lgssc registered [ 6865.034872] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6867.915851] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6867.927899] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6869.039477] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6869.048296] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6869.105598] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6870.301116] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 6873.298606] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6873.330287] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6873.342339] Lustre: 362364:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6873.437796] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6874.832743] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 6876.636237] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6877.809638] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6878.689127] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6878.691919] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6878.696824] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6879.713483] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6879.713859] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6884.625708] Lustre: server umount lustre-MDT0000 complete [ 6885.813066] LustreError: 363087:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777503624 with bad export cookie 14675680294383700402 [ 6885.815147] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6885.816657] LustreError: 363087:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6885.927854] Lustre: server umount lustre-MDT0001 complete [ 6889.068137] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 19:00:26 (1777503626) [ 6891.237382] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6891.391130] LustreError: 364645:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6891.397536] LustreError: 364645:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 6891.408815] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6892.696233] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 6895.465824] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6895.582164] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6896.792927] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 6898.606320] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6899.792446] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6901.964300] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6902.033379] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6902.089512] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6903.822487] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 6905.492454] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6914.328496] Lustre: Permanently deactivating lustre-OST0000 [ 6914.333357] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 6922.231161] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6922.891487] LustreError: 367397:0:(mgs_llog.c:2031:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [ 6922.895327] LustreError: 367397:0:(mgs_llog.c:2086:mgs_replace_nids()) Only MGS is allowed to be started [ 6922.897774] LustreError: 367397:0:(mgs_handler.c:1244:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 6923.491401] LustreError: 367445:0:(mgs_llog.c:2086:mgs_replace_nids()) Only MGS is allowed to be started [ 6923.493822] LustreError: 367445:0:(mgs_handler.c:1244:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 6930.264378] Lustre: server umount lustre-OST0000 complete [ 6931.218617] LustreError: 367695:0:(mgs_llog.c:2086:mgs_replace_nids()) Only MGS is allowed to be started [ 6931.221565] LustreError: 367695:0:(mgs_handler.c:1244:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 6936.545812] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6936.546053] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6936.546695] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6936.546700] Lustre: Skipped 1 previous similar message [ 6936.558058] Lustre: Skipped 1 previous similar message [ 6941.665049] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6941.667044] Lustre: Skipped 2 previous similar messages [ 6946.272108] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6946.334116] Lustre: server umount lustre-MDT0000 complete [ 6946.784898] LustreError: 364645:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6946.789361] LustreError: 364645:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 6947.488131] LustreError: 365360:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777503686 with bad export cookie 14675680294383700969 [ 6947.493676] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6947.494199] LustreError: 365360:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6958.088905] Lustre: server umount lustre-MDT0001 complete [ 6960.785416] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6962.065079] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 6962.991365] LustreError: 368837:0:(mgs_handler.c:1244:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [ 6965.048388] Lustre: 369175:0:(mgs_llog.c:1798:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.106@tcp [ 6965.721988] Lustre: server umount MGS complete [ 6969.540497] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6969.714973] LustreError: 369898:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6969.719288] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6969.722543] Lustre: Skipped 1 previous similar message [ 6969.743595] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6971.032434] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 6973.784765] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6973.882975] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6975.072514] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 6976.815350] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6978.027372] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6978.693109] Lustre: Permanently reactivating lustre-OST0000 [ 6978.699831] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 6978.701838] Lustre: Skipped 2 previous similar messages [ 7003.104138] LustreError: 371404:0:(osp_dev.c:695:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [ 7006.209177] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7006.320519] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7006.322486] Lustre: Skipped 1 previous similar message [ 7008.102869] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 7010.416877] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7023.073731] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7023.077779] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7023.078339] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7023.079685] Lustre: Skipped 1 previous similar message [ 7023.082478] LustreError: Skipped 1 previous similar message [ 7026.131154] Lustre: server umount lustre-OST0000 complete [ 7028.195125] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7028.202778] Lustre: Skipped 1 previous similar message [ 7033.312991] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7033.314775] Lustre: Skipped 4 previous similar messages [ 7033.553658] Lustre: server umount lustre-MDT0000 complete [ 7035.108632] LustreError: 369924:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777503773 with bad export cookie 14675680294383702005 [ 7035.114122] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7035.114730] LustreError: 369924:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7035.254556] Lustre: server umount lustre-MDT0001 complete [ 7038.903509] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 7039.882226] Key type lgssc unregistered [ 7040.011352] LNet: 374145:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7040.013879] LNetError: 374145:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7040.021295] LNet: Removed LNI 192.168.204.106@tcp [ 7040.303424] Key type .llcrypt unregistered [ 7040.304722] Key type ._llcrypt unregistered [ 7046.124323] Key type ._llcrypt registered [ 7046.125424] Key type .llcrypt registered [ 7046.168045] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_hostid [ 7052.908240] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 7053.305486] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7053.375463] alg: No test for adler32 (adler32-zlib) [ 7054.268536] Lustre: Lustre: Build Version: 2.17.52_93_g6cdb58a [ 7054.360738] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 7055.944150] Key type lgssc registered [ 7056.304428] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7068.422936] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7068.428505] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7069.516027] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7069.526271] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7069.563133] Lustre: lustre-MDT0000: new disk, initializing [ 7069.585772] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7069.592147] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7070.866507] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 7075.335742] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7075.380540] Lustre: 377240:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7075.405275] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7075.407078] Lustre: Skipped 1 previous similar message [ 7075.445623] Lustre: lustre-MDT0001: new disk, initializing [ 7075.475772] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7075.487855] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7075.492022] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7076.775551] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 7079.944597] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7081.169602] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7083.497719] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7083.618153] Lustre: lustre-OST0000: new disk, initializing [ 7083.619855] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7083.647220] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7085.526634] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 7086.775343] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7086.778390] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7086.809644] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7088.587612] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7089.961530] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7090.034302] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7091.327897] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7091.405272] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7092.196247] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7092.196901] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7092.198217] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7092.201699] LustreError: Skipped 1 previous similar message [ 7092.207567] Lustre: Skipped 1 previous similar message [ 7097.315572] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7097.318216] Lustre: Skipped 2 previous similar messages [ 7098.201554] Lustre: server umount lustre-OST0000 complete [ 7101.409432] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7101.414617] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7101.423881] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7102.437191] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7105.772872] Lustre: server umount lustre-MDT0000 complete [ 7107.033420] LustreError: 377233:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777503845 with bad export cookie 18200890617418763223 [ 7107.036403] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7107.038029] LustreError: 377233:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7107.174093] Lustre: server umount lustre-MDT0001 complete [ 7109.985871] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [ 7110.576451] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 19:04:08 (1777503848) [ 7113.167267] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7113.352521] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7114.716430] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 7117.914716] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7118.042575] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7119.579684] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 7121.319470] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7122.595722] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7124.881943] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7126.668991] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 7128.342960] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7128.963639] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [ 7139.858198] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [ 7143.393399] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7143.397407] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7143.405131] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7143.408239] Lustre: Skipped 2 previous similar messages [ 7148.710164] Lustre: server umount lustre-OST0000 complete [ 7154.145238] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7154.150676] LustreError: Skipped 1 previous similar message [ 7154.155430] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7154.164348] Lustre: Skipped 1 previous similar message [ 7154.167180] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7154.171320] Lustre: Skipped 2 previous similar messages [ 7156.485143] Lustre: server umount lustre-MDT0000 complete [ 7158.499094] LustreError: 381271:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777503897 with bad export cookie 18200890617418764098 [ 7158.501783] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7158.503858] LustreError: 381271:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7158.684132] Lustre: server umount lustre-MDT0001 complete [ 7165.884205] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 7168.019030] Key type lgssc unregistered [ 7168.305453] LNet: 384676:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7168.312198] LNetError: 384676:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7168.330818] LNet: Removed LNI 192.168.204.106@tcp [ 7169.145170] Key type .llcrypt unregistered [ 7169.147985] Key type ._llcrypt unregistered [ 7187.872509] Key type ._llcrypt registered [ 7187.874713] Key type .llcrypt registered [ 7187.926844] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 7188.069709] Key type .llcrypt unregistered [ 7188.071821] Key type ._llcrypt unregistered [ 7197.262199] Key type ._llcrypt registered [ 7197.264119] Key type .llcrypt registered [ 7197.306780] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 7197.765147] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7197.802694] alg: No test for adler32 (adler32-zlib) [ 7198.720456] Lustre: Lustre: Build Version: 2.17.52_93_g6cdb58a [ 7198.828781] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 7200.424263] Key type lgssc registered [ 7200.977165] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7204.794703] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7204.812674] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7206.022505] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7207.347379] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 7210.407093] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7210.587669] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7212.919050] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 7214.913234] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7215.587370] LustreError: 386831:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7215.593532] LustreError: 386831:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 4 previous similar messages [ 7216.610594] LustreError: 386830:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7216.617333] LustreError: 386830:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 7216.627149] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 7217.550746] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7217.738806] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7219.371542] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 7227.874931] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7227.877327] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7227.885295] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7232.995524] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7232.999045] Lustre: Skipped 2 previous similar messages [ 7233.388878] Lustre: server umount lustre-OST0000 complete [ 7238.114873] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7238.115396] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7238.116624] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7238.130813] Lustre: Skipped 2 previous similar messages [ 7241.336457] Lustre: server umount lustre-MDT0000 complete [ 7242.871113] LustreError: 388788:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777503981 with bad export cookie 4147314723253427996 [ 7242.873803] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7242.876892] LustreError: 388788:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7243.007081] Lustre: server umount lustre-MDT0001 complete [ 7246.983359] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 7248.157630] Key type lgssc unregistered [ 7248.296436] LNet: 389810:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7248.299952] LNetError: 389810:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7248.307445] LNet: Removed LNI 192.168.204.106@tcp [ 7248.622844] Key type .llcrypt unregistered [ 7248.624469] Key type ._llcrypt unregistered [ 7258.294170] Key type ._llcrypt registered [ 7258.296356] Key type .llcrypt registered [ 7258.348763] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 7258.699329] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7258.760174] alg: No test for adler32 (adler32-zlib) [ 7259.641112] Lustre: Lustre: Build Version: 2.17.52_93_g6cdb58a [ 7259.748434] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 7261.352297] Key type lgssc registered [ 7261.782353] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7264.855848] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7264.863861] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7280.289552] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7285.408509] LustreError: 391161:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.106@tcp: failed processing log, type 4: rc = -110 [ 7312.032179] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7314.818030] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 7317.589131] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7321.154505] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7321.411522] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7321.463233] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 7323.579373] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 7326.501506] LustreError: 392086:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7326.515291] LustreError: 392086:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 7327.974684] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7328.189036] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7330.370249] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 7333.251798] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7335.344972] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7342.050988] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7342.053682] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7342.059995] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7343.587224] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7343.590014] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7343.593730] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7343.603879] Lustre: Skipped 1 previous similar message [ 7345.987803] Lustre: server umount lustre-OST0000 complete [ 7348.706521] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7348.706794] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7348.715987] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7348.724346] Lustre: Skipped 1 previous similar message [ 7350.134773] Lustre: server umount lustre-MDT0000 complete [ 7352.094120] LustreError: 391168:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777504090 with bad export cookie 14588814331750903851 [ 7352.098923] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7352.101696] LustreError: 391168:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7352.296183] Lustre: server umount lustre-MDT0001 complete [ 7357.901650] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 7359.543251] Key type lgssc unregistered [ 7359.752127] LNet: 394371:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7359.762805] LNetError: 394371:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7359.781901] LNet: Removed LNI 192.168.204.106@tcp [ 7360.258221] Key type .llcrypt unregistered [ 7360.260827] Key type ._llcrypt unregistered [ 7373.968233] Key type ._llcrypt registered [ 7373.969620] Key type .llcrypt registered [ 7374.041968] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 7374.530617] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7374.565198] alg: No test for adler32 (adler32-zlib) [ 7375.506877] Lustre: Lustre: Build Version: 2.17.52_93_g6cdb58a [ 7375.644915] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 7377.272239] Key type lgssc registered [ 7377.925803] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7382.383931] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7382.406106] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7383.670883] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7385.760339] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 7390.718945] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7390.928201] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7393.150550] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 7395.975435] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7397.829406] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7401.468077] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7401.733092] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7405.574782] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 7408.671505] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7408.942533] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 7414.246632] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7414.249373] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7414.252436] Lustre: Skipped 2 previous similar messages [ 7414.258078] Lustre: Skipped 2 previous similar messages [ 7416.237972] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.6@tcp (stopping) [ 7418.470069] Lustre: server umount lustre-MDT0000 complete [ 7419.991076] LustreError: 395728:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.6@tcp arrived at 1777504158 with bad export cookie 12703750617914147409 [ 7421.220576] LustreError: 395727:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777504160 with bad export cookie 12703750617914146520 [ 7421.227365] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7421.228074] LustreError: 395727:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7421.411740] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7421.413587] LustreError: 395744:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7421.425694] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7421.433072] LustreError: 395744:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 7421.443903] Lustre: Skipped 3 previous similar messages [ 7423.351526] Lustre: server umount lustre-OST0000 complete [ 7426.530713] LustreError: 395747:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7426.546445] LustreError: 395747:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 7428.640705] LustreError: 396458:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000002ff5ac41 x1863848153867520/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7428.752626] Lustre: server umount lustre-MDT0001 complete [ 7434.793467] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 7436.797619] Key type lgssc unregistered [ 7437.011886] LNet: 399069:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7437.022220] LNetError: 399069:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7437.037481] LNet: Removed LNI 192.168.204.106@tcp [ 7437.597437] Key type .llcrypt unregistered [ 7437.599690] Key type ._llcrypt unregistered [ 7453.211564] Key type ._llcrypt registered [ 7453.213639] Key type .llcrypt registered [ 7453.312585] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 7454.070650] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7454.137588] alg: No test for adler32 (adler32-zlib) [ 7455.145487] Lustre: Lustre: Build Version: 2.17.52_93_g6cdb58a [ 7455.284113] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 7456.929354] Key type lgssc registered [ 7457.667655] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7462.385604] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7462.398115] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7463.682974] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7465.679246] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 7469.643665] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7469.836796] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7471.834238] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 7474.405815] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7476.469120] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7480.019933] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7480.100191] LustreError: 402042:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7480.109340] LustreError: 402042:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 7480.220754] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7483.323778] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 7486.226746] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7488.491049] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 7494.625375] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 7494.628920] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7494.635174] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7497.633995] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.6@tcp (stopping) [ 7498.722060] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7498.728091] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7499.919442] Lustre: server umount lustre-MDT0001 complete [ 7503.843718] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7503.844804] LustreError: 400449:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7503.845508] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7503.853074] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7503.876869] Lustre: Skipped 1 previous similar message [ 7508.961526] LustreError: 401557:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7508.962917] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7509.296553] Lustre: server umount lustre-OST0000 complete [ 7511.499299] Lustre: server umount lustre-MDT0000 complete [ 7518.650912] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 7520.153303] Key type lgssc unregistered [ 7520.339640] LNet: 403775:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7520.344632] LNetError: 403775:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7520.355187] LNet: Removed LNI 192.168.204.106@tcp [ 7520.821466] Key type .llcrypt unregistered [ 7520.823346] Key type ._llcrypt unregistered [ 7534.613281] Key type ._llcrypt registered [ 7534.615258] Key type .llcrypt registered [ 7534.670721] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_hostid [ 7543.335958] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 7543.874933] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7543.919599] alg: No test for adler32 (adler32-zlib) [ 7544.864124] Lustre: Lustre: Build Version: 2.17.52_93_g6cdb58a [ 7545.006946] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 7546.608169] Key type lgssc registered [ 7547.279830] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7567.005248] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 7575.948649] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7575.970734] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7577.121543] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7577.145377] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7577.190747] Lustre: lustre-MDT0000: new disk, initializing [ 7577.233959] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7577.259322] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7579.825116] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 7589.124151] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7589.293433] Lustre: lustre-OST0000: new disk, initializing [ 7589.295053] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 7589.299335] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7589.308850] Lustre: Skipped 1 previous similar message [ 7589.350685] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7592.822444] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 7594.498149] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 7594.508252] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 7594.529893] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 7597.396844] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7606.582135] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7606.638397] Lustre: 407715:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7606.678924] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7606.751196] Lustre: lustre-MDT0001: new disk, initializing [ 7606.826428] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7606.857497] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 7606.865890] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 7608.938676] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 7614.171452] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7614.284412] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7616.616975] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7616.749496] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7620.512668] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.6@tcp (stopping) [ 7621.548183] Lustre: server umount lustre-MDT0001 complete [ 7624.164472] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 7624.165315] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7624.176311] Lustre: Skipped 1 previous similar message [ 7629.285823] LustreError: 407719:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7629.297287] LustreError: 407719:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 4 previous similar messages [ 7634.403987] LustreError: 407721:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7634.413435] LustreError: 407721:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 7635.427597] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7635.430463] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7635.437422] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7639.523808] LustreError: 407724:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7639.527316] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7644.642373] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7644.643514] LustreError: 407725:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7649.248127] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7649.336355] Lustre: server umount lustre-OST0000 complete [ 7649.762360] LustreError: 407724:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7657.249665] Lustre: server umount lustre-MDT0000 complete [ 7663.625650] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 7665.063821] Key type lgssc unregistered [ 7665.233528] LNet: 412275:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7665.237555] LNetError: 412275:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7665.246825] LNet: Removed LNI 192.168.204.106@tcp [ 7665.641177] Key type .llcrypt unregistered [ 7665.642929] Key type ._llcrypt unregistered [ 7684.246894] Key type ._llcrypt registered [ 7684.248882] Key type .llcrypt registered [ 7684.307060] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 7684.474258] Key type .llcrypt unregistered [ 7684.477761] Key type ._llcrypt unregistered [ 7692.965354] Key type ._llcrypt registered [ 7692.966500] Key type .llcrypt registered [ 7693.018417] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 7693.436090] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7693.444152] alg: No test for adler32 (adler32-zlib) [ 7694.331789] Lustre: Lustre: Build Version: 2.17.52_93_g6cdb58a [ 7694.457096] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 7696.056127] Key type lgssc registered [ 7696.620653] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7709.163873] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7709.172373] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7710.306407] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7710.328842] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7710.369824] Lustre: lustre-MDT0000: new disk, initializing [ 7710.429089] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7710.441299] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7712.369437] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 7718.120540] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7718.175901] Lustre: 415606:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7718.212496] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7718.215417] Lustre: Skipped 1 previous similar message [ 7718.263817] Lustre: lustre-MDT0001: new disk, initializing [ 7718.310448] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7718.329823] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7718.335309] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7720.105917] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 7724.152639] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7725.984616] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7729.428801] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7729.570632] Lustre: lustre-OST0000: new disk, initializing [ 7729.573863] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7729.625626] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7732.488889] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 7735.312925] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7735.318041] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7735.368130] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7736.644831] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7740.386984] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7740.387710] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7740.402031] Lustre: Skipped 2 previous similar messages [ 7740.405473] Lustre: Skipped 1 previous similar message [ 7745.505631] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7745.509208] Lustre: Skipped 2 previous similar messages [ 7746.162471] Lustre: server umount lustre-MDT0000 complete [ 7747.787550] LustreError: 415598:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777504486 with bad export cookie 8186911679238700419 [ 7747.793406] LustreError: 415598:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7747.793608] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7747.965705] Lustre: server umount lustre-MDT0001 complete [ 7759.985040] Lustre: server umount lustre-OST0000 complete [ 7764.772029] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 19:15:02 (1777504502) [ 7768.537525] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7768.810966] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7770.673636] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 7774.179340] LustreError: 419261:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7774.189245] LustreError: 419261:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 7774.777469] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7774.966864] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7776.765507] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 7779.382916] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7781.109644] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7797.414088] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7797.538581] Lustre: Found index 0 for lustre-OST0000, updating log [ 7797.645293] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7800.597446] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 7803.436984] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7815.082265] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [ 7821.281553] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7821.285263] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7821.291713] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7826.856971] Lustre: server umount lustre-OST0000 complete [ 7841.761122] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7841.765536] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7841.772296] Lustre: Skipped 1 previous similar message [ 7841.776260] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7841.779123] Lustre: Skipped 3 previous similar messages [ 7843.809750] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7846.202249] Lustre: server umount lustre-MDT0000 complete [ 7848.077037] LustreError: 419241:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777504586 with bad export cookie 8186911679238701805 [ 7848.080482] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7848.085676] LustreError: 419241:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7848.272274] Lustre: server umount lustre-MDT0001 complete [ 7852.013875] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 19:16:29 (1777504589) [ 7855.264227] LNet: 1 local NIs in recovery (showing 1): 192.168.204.106@tcp [ 8433.208925] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8433.348352] LustreError: 425342:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8433.367603] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8434.642934] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 8437.294145] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8437.408236] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8438.653730] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 8440.345204] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8441.445124] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8456.547142] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8456.623127] Lustre: Found index 0 for lustre-OST0000, updating log [ 8456.713356] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8458.446984] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 8460.146450] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8460.832430] Lustre: server umount lustre-OST0000 complete [ 8463.328685] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8463.331040] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8463.334894] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8463.336710] Lustre: Skipped 2 previous similar messages [ 8464.353149] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8464.353660] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8465.377115] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8465.379621] Lustre: Skipped 2 previous similar messages [ 8468.118422] Lustre: server umount lustre-MDT0000 complete [ 8469.334132] LustreError: 425322:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777505208 with bad export cookie 8186911679238702883 [ 8469.337035] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8469.338454] LustreError: 425322:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8469.452623] Lustre: server umount lustre-MDT0001 complete [ 8486.702126] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_hostid [ 8490.146578] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 8495.264295] LNet: 1 local NIs in recovery (showing 1): 192.168.204.106@tcp [ 8502.327245] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 19:27:20 (1777505240) [ 8507.720834] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 8508.654983] Key type lgssc unregistered [ 8519.072648] LNet: 431601:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8519.074984] LNetError: 431601:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8586.662308] LNet: Removed LNI 192.168.204.106@tcp [ 8586.965416] Key type .llcrypt unregistered [ 8586.967043] Key type ._llcrypt unregistered [ 8593.747476] Key type ._llcrypt registered [ 8593.748666] Key type .llcrypt registered [ 8593.790171] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 8594.122738] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8594.131956] alg: No test for adler32 (adler32-zlib) [ 8594.991260] Lustre: Lustre: Build Version: 2.17.52_93_g6cdb58a [ 8595.074763] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 8596.664153] Key type lgssc registered [ 8597.052524] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8598.527790] LNet: Added LNI 192.168.204.106@tcp100 [8/256/0/180] [ 8598.831429] LNet: Added LNI 192.168.204.106@tcp101 [8/256/0/180] [ 8599.127976] LNet: Added LNI 192.168.204.106@tcp102 [8/256/0/180] [ 8599.417919] LNet: Added LNI 192.168.204.106@tcp103 [8/256/0/180] [ 8599.701163] LNet: Added LNI 192.168.204.106@tcp104 [8/256/0/180] [ 8599.990182] LNet: Added LNI 192.168.204.106@tcp105 [8/256/0/180] [ 8600.284806] LNet: Added LNI 192.168.204.106@tcp106 [8/256/0/180] [ 8600.578026] LNet: Added LNI 192.168.204.106@tcp107 [8/256/0/180] [ 8600.879111] LNet: Added LNI 192.168.204.106@tcp108 [8/256/0/180] [ 8601.176224] LNet: Added LNI 192.168.204.106@tcp109 [8/256/0/180] [ 8601.483199] LNet: Added LNI 192.168.204.106@tcp110 [8/256/0/180] [ 8601.782098] LNet: Added LNI 192.168.204.106@tcp111 [8/256/0/180] [ 8602.097534] LNet: Added LNI 192.168.204.106@tcp112 [8/256/0/180] [ 8602.408920] LNet: Added LNI 192.168.204.106@tcp113 [8/256/0/180] [ 8602.705898] LNet: Added LNI 192.168.204.106@tcp114 [8/256/0/180] [ 8603.004692] LNet: Added LNI 192.168.204.106@tcp115 [8/256/0/180] [ 8603.309583] LNet: Added LNI 192.168.204.106@tcp116 [8/256/0/180] [ 8603.620887] LNet: Added LNI 192.168.204.106@tcp117 [8/256/0/180] [ 8603.927362] LNet: Added LNI 192.168.204.106@tcp118 [8/256/0/180] [ 8604.222509] LNet: Added LNI 192.168.204.106@tcp119 [8/256/0/180] [ 8604.513298] LNet: Added LNI 192.168.204.106@tcp120 [8/256/0/180] [ 8604.806277] LNet: Added LNI 192.168.204.106@tcp121 [8/256/0/180] [ 8605.084348] LNet: Added LNI 192.168.204.106@tcp122 [8/256/0/180] [ 8605.366341] LNet: Added LNI 192.168.204.106@tcp123 [8/256/0/180] [ 8605.657326] LNet: Added LNI 192.168.204.106@tcp124 [8/256/0/180] [ 8605.940378] LNet: Added LNI 192.168.204.106@tcp125 [8/256/0/180] [ 8606.239740] LNet: Added LNI 192.168.204.106@tcp126 [8/256/0/180] [ 8606.555372] LNet: Added LNI 192.168.204.106@tcp127 [8/256/0/180] [ 8606.857703] LNet: Added LNI 192.168.204.106@tcp128 [8/256/0/180] [ 8607.144706] LNet: Added LNI 192.168.204.106@tcp129 [8/256/0/180] [ 8607.429735] LNet: Added LNI 192.168.204.106@tcp130 [8/256/0/180] [ 8607.723974] LNet: Added LNI 192.168.204.106@tcp131 [8/256/0/180] [ 8608.015222] LNet: Added LNI 192.168.204.106@tcp132 [8/256/0/180] [ 8608.315477] LNet: Added LNI 192.168.204.106@tcp133 [8/256/0/180] [ 8608.608900] LNet: Added LNI 192.168.204.106@tcp134 [8/256/0/180] [ 8608.898745] LNet: Added LNI 192.168.204.106@tcp135 [8/256/0/180] [ 8611.090275] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8611.097962] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8612.135504] Lustre: 434583:0:(lustre_peer.c:105:class_add_uuid()) 192.168.204.106@tcp: can't add NID 192.168.204.106@tcp130: rc = -75 [ 8612.194950] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8612.206355] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8612.251039] Lustre: lustre-MDT0000: new disk, initializing [ 8612.279378] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8612.285743] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8613.605700] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 8617.697929] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8617.744276] Lustre: 434600:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8617.771518] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8617.774573] Lustre: Skipped 1 previous similar message [ 8617.811262] Lustre: lustre-MDT0001: new disk, initializing [ 8617.823318] Lustre: 435379:0:(lustre_peer.c:105:class_add_uuid()) 192.168.204.106@tcp: can't add NID 192.168.204.106@tcp130: rc = -75 [ 8617.826262] Lustre: 435379:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [ 8617.861723] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8617.872419] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8617.875865] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8619.092485] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 8622.152284] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8623.396618] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8625.617556] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8625.731016] Lustre: lustre-OST0000: new disk, initializing [ 8625.733858] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8625.767647] Lustre: 436344:0:(lustre_peer.c:105:class_add_uuid()) 192.168.204.106@tcp: can't add NID 192.168.204.106@tcp130: rc = -75 [ 8625.771466] Lustre: 436344:0:(lustre_peer.c:105:class_add_uuid()) Skipped 17 previous similar messages [ 8625.787639] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8626.947912] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8626.951992] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8626.986412] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8627.661264] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 8630.731133] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8637.410937] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8637.411079] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8637.415503] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8637.419423] Lustre: Skipped 1 previous similar message [ 8639.115780] Lustre: server umount lustre-OST0000 complete [ 8642.529733] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8642.530484] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8642.533663] Lustre: Skipped 2 previous similar messages [ 8646.428432] Lustre: server umount lustre-MDT0000 complete [ 8647.619118] LustreError: 435378:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777505386 with bad export cookie 14100889087908000116 [ 8647.621722] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8647.624586] LustreError: 435378:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8647.740873] Lustre: server umount lustre-MDT0001 complete [ 8651.232032] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 8652.249691] Key type lgssc unregistered [ 8652.416237] LNet: 438123:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8652.420378] LNetError: 438123:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8652.424539] LNet: Removed LNI 192.168.204.106@tcp [ 8653.472180] LNet: Removed LNI 192.168.204.106@tcp100 [ 8653.473627] LNetError: 438123:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8653.475681] LNetError: 438123:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 8653.477885] LNet: Removed LNI 192.168.204.106@tcp101 [ 8653.479061] LNet: Removed LNI 192.168.204.106@tcp102 [ 8653.480257] LNet: Removed LNI 192.168.204.106@tcp103 [ 8653.481470] LNet: Removed LNI 192.168.204.106@tcp104 [ 8653.482595] LNet: Removed LNI 192.168.204.106@tcp105 [ 8653.483689] LNet: Removed LNI 192.168.204.106@tcp106 [ 8653.485312] LNet: Removed LNI 192.168.204.106@tcp107 [ 8653.487100] LNet: Removed LNI 192.168.204.106@tcp108 [ 8653.488793] LNet: Removed LNI 192.168.204.106@tcp109 [ 8653.490529] LNet: Removed LNI 192.168.204.106@tcp110 [ 8653.492302] LNet: Removed LNI 192.168.204.106@tcp111 [ 8653.494053] LNet: Removed LNI 192.168.204.106@tcp112 [ 8653.495173] LNet: Removed LNI 192.168.204.106@tcp113 [ 8653.496317] LNet: Removed LNI 192.168.204.106@tcp114 [ 8653.497517] LNet: Removed LNI 192.168.204.106@tcp115 [ 8653.498981] LNet: Removed LNI 192.168.204.106@tcp116 [ 8653.500423] LNet: Removed LNI 192.168.204.106@tcp117 [ 8653.501703] LNet: Removed LNI 192.168.204.106@tcp118 [ 8653.503264] LNet: Removed LNI 192.168.204.106@tcp119 [ 8653.504531] LNet: Removed LNI 192.168.204.106@tcp120 [ 8653.505742] LNet: Removed LNI 192.168.204.106@tcp121 [ 8653.506962] LNet: Removed LNI 192.168.204.106@tcp122 [ 8653.508797] LNet: Removed LNI 192.168.204.106@tcp123 [ 8653.509938] LNet: Removed LNI 192.168.204.106@tcp124 [ 8653.511637] LNet: Removed LNI 192.168.204.106@tcp125 [ 8653.512786] LNet: Removed LNI 192.168.204.106@tcp126 [ 8653.514096] LNet: Removed LNI 192.168.204.106@tcp127 [ 8653.515456] LNet: Removed LNI 192.168.204.106@tcp128 [ 8653.516603] LNet: Removed LNI 192.168.204.106@tcp129 [ 8653.518198] LNet: Removed LNI 192.168.204.106@tcp130 [ 8653.519347] LNet: Removed LNI 192.168.204.106@tcp131 [ 8653.520855] LNet: Removed LNI 192.168.204.106@tcp132 [ 8653.522235] LNet: Removed LNI 192.168.204.106@tcp133 [ 8653.523386] LNet: Removed LNI 192.168.204.106@tcp134 [ 8653.531316] LNet: Removed LNI 192.168.204.106@tcp135 [ 8653.791645] Key type .llcrypt unregistered [ 8653.792765] Key type ._llcrypt unregistered [ 8661.309656] Key type ._llcrypt registered [ 8661.310734] Key type .llcrypt registered [ 8661.346759] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_hostid [ 8667.626757] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 8667.959248] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8667.991062] alg: No test for adler32 (adler32-zlib) [ 8668.843192] Lustre: Lustre: Build Version: 2.17.52_93_g6cdb58a [ 8668.929862] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 8670.512121] Key type lgssc registered [ 8670.855504] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8682.042876] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 19:30:19 (1777505419) [ 8688.166929] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 8689.194878] Key type lgssc unregistered [ 8689.318338] LNet: 442128:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8689.321185] LNetError: 442128:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8689.327282] LNet: Removed LNI 192.168.204.106@tcp [ 8689.635433] Key type .llcrypt unregistered [ 8689.637458] Key type ._llcrypt unregistered [ 8696.070297] Key type ._llcrypt registered [ 8696.071428] Key type .llcrypt registered [ 8696.105510] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 8696.543664] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8696.551510] alg: No test for adler32 (adler32-zlib) [ 8697.406861] Lustre: Lustre: Build Version: 2.17.52_93_g6cdb58a [ 8697.485052] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 8699.064142] Key type lgssc registered [ 8699.427088] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8702.508288] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8702.513391] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8703.600697] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8703.613067] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8703.655103] Lustre: lustre-MDT0000: new disk, initializing [ 8703.680175] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8703.687528] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8704.982911] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 8709.161579] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8709.205263] Lustre: 443299:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8709.228644] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8709.231553] Lustre: Skipped 1 previous similar message [ 8709.267823] Lustre: lustre-MDT0001: new disk, initializing [ 8709.302296] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8709.314980] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8709.319527] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8710.650155] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 8713.773580] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8714.969356] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8717.128772] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8717.231845] Lustre: lustre-OST0000: new disk, initializing [ 8717.234213] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8717.261665] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8719.255782] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 8721.912050] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8721.916500] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8721.942835] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8722.357389] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8723.605731] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing lnet_if_list [ 8725.213372] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [ 8725.225588] LNet: Added LNI 192.168.204.106@tcp10 [8/256/0/180] [ 8726.922320] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [ 8726.937703] LNet: Added LNI 192.168.204.106@tcp11 [8/256/0/180] [ 8728.234190] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [ 8728.245927] LNet: Added LNI 192.168.204.106@tcp12 [8/256/0/180] [ 8729.633700] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [ 8729.646554] LNet: Added LNI 192.168.204.106@tcp13 [8/256/0/180] [ 8731.089474] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [ 8731.102723] LNet: Added LNI 192.168.204.106@tcp14 [8/256/0/180] [ 8732.599086] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [ 8732.613181] LNet: Added LNI 192.168.204.106@tcp15 [8/256/0/180] [ 8734.093475] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [ 8734.106304] LNet: Added LNI 192.168.204.106@tcp16 [8/256/0/180] [ 8735.650654] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [ 8735.663693] LNet: Added LNI 192.168.204.106@tcp17 [8/256/0/180] [ 8737.206277] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [ 8737.218430] LNet: Added LNI 192.168.204.106@tcp18 [8/256/0/180] [ 8738.747575] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [ 8738.759358] LNet: Added LNI 192.168.204.106@tcp19 [8/256/0/180] [ 8740.361860] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [ 8740.372694] LNet: Added LNI 192.168.204.106@tcp20 [8/256/0/180] [ 8741.978613] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [ 8741.989223] LNet: Added LNI 192.168.204.106@tcp21 [8/256/0/180] [ 8743.610275] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [ 8743.623451] LNet: Added LNI 192.168.204.106@tcp22 [8/256/0/180] [ 8745.284929] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [ 8745.297367] LNet: Added LNI 192.168.204.106@tcp23 [8/256/0/180] [ 8747.087988] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [ 8747.098465] LNet: Added LNI 192.168.204.106@tcp24 [8/256/0/180] [ 8748.902550] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [ 8748.914699] LNet: Added LNI 192.168.204.106@tcp25 [8/256/0/180] [ 8750.769462] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [ 8750.781830] LNet: Added LNI 192.168.204.106@tcp26 [8/256/0/180] [ 8752.760825] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [ 8752.772051] LNet: Added LNI 192.168.204.106@tcp27 [8/256/0/180] [ 8754.503111] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [ 8754.513654] LNet: Added LNI 192.168.204.106@tcp28 [8/256/0/180] [ 8756.353711] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [ 8756.365292] LNet: Added LNI 192.168.204.106@tcp29 [8/256/0/180] [ 8758.204294] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [ 8758.218228] LNet: Added LNI 192.168.204.106@tcp30 [8/256/0/180] [ 8760.079811] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [ 8760.091968] LNet: Added LNI 192.168.204.106@tcp31 [8/256/0/180] [ 8762.078201] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [ 8762.088232] LNet: Added LNI 192.168.204.106@tcp32 [8/256/0/180] [ 8764.131697] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [ 8764.142969] LNet: Added LNI 192.168.204.106@tcp33 [8/256/0/180] [ 8766.298609] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [ 8766.310908] LNet: Added LNI 192.168.204.106@tcp34 [8/256/0/180] [ 8768.490846] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [ 8768.504567] LNet: Added LNI 192.168.204.106@tcp35 [8/256/0/180] [ 8770.685177] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [ 8770.697750] LNet: Added LNI 192.168.204.106@tcp36 [8/256/0/180] [ 8772.881496] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [ 8772.893442] LNet: Added LNI 192.168.204.106@tcp37 [8/256/0/180] [ 8775.106236] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [ 8775.120363] LNet: Added LNI 192.168.204.106@tcp38 [8/256/0/180] [ 8777.284978] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [ 8777.296938] LNet: Added LNI 192.168.204.106@tcp39 [8/256/0/180] [ 8779.619869] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [ 8779.634612] LNet: Added LNI 192.168.204.106@tcp40 [8/256/0/180] [ 8782.128753] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [ 8782.141636] LNet: Added LNI 192.168.204.106@tcp41 [8/256/0/180] [ 8784.796561] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [ 8784.812947] LNet: Added LNI 192.168.204.106@tcp42 [8/256/0/180] [ 8787.638171] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [ 8787.650898] LNet: Added LNI 192.168.204.106@tcp43 [8/256/0/180] [ 8790.496768] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [ 8790.509654] LNet: Added LNI 192.168.204.106@tcp44 [8/256/0/180] [ 8793.129836] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [ 8793.144546] LNet: Added LNI 192.168.204.106@tcp45 [8/256/0/180] [ 8795.814162] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [ 8795.828222] LNet: Added LNI 192.168.204.106@tcp46 [8/256/0/180] [ 8798.585436] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [ 8798.599842] LNet: Added LNI 192.168.204.106@tcp47 [8/256/0/180] [ 8801.706109] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [ 8801.722102] LNet: Added LNI 192.168.204.106@tcp48 [8/256/0/180] [ 8804.431068] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [ 8804.444713] LNet: Added LNI 192.168.204.106@tcp49 [8/256/0/180] [ 8807.282146] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [ 8807.297481] LNet: Added LNI 192.168.204.106@tcp50 [8/256/0/180] [ 8817.121035] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8817.123557] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8817.128841] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8819.169712] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8819.172867] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8820.830477] Lustre: server umount lustre-MDT0000 complete [ 8822.162115] LustreError: 444078:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777505560 with bad export cookie 9362004214211812932 [ 8822.165461] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8822.166528] LustreError: 444078:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8822.290038] Lustre: server umount lustre-MDT0001 complete [ 8829.284700] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 8832.478319] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8840.672154] Lustre: 442573:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1777505563/real 1777505563] req@0000000011c55f09 x1863849540141824/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1777505579 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8840.679922] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8840.683516] Lustre: Skipped 1 previous similar message [ 8843.424135] Lustre: 442574:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1777505566/real 1777505566] req@00000000d895cd92 x1863849540141952/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1777505582 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8845.792165] Lustre: 442575:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1777505568/real 1777505568] req@00000000757a3f1f x1863849540142080/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1777505584 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8846.816492] LustreError: 442572:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000b9a799bd x1863849540142976/t0(0) o250->MGC192.168.204.106@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8846.973313] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8848.032090] Lustre: 442576:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1777505570/real 1777505570] req@00000000f51afad7 x1863849540142336/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1777505586 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8848.455789] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 8851.561305] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8851.699993] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8852.704091] Lustre: 442573:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1777505575/real 1777505575] req@0000000012589117 x1863849540142464/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1777505591 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8853.027875] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 8854.687962] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8855.834407] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8857.061954] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 8860.134262] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 8865.250706] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8865.251793] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8865.255251] Lustre: Skipped 1 previous similar message [ 8865.257535] Lustre: Skipped 2 previous similar messages [ 8869.391976] Lustre: server umount lustre-OST0000 complete [ 8872.416869] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8872.419798] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8872.423476] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8876.764053] Lustre: server umount lustre-MDT0000 complete [ 8878.022911] LustreError: 443292:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777505616 with bad export cookie 9362004214211814997 [ 8878.026082] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8878.028121] LustreError: 443292:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8878.167562] Lustre: server umount lustre-MDT0001 complete [ 8881.909915] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 8882.935627] Key type lgssc unregistered [ 8883.090690] LNet: 470915:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8883.093895] LNetError: 470915:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8883.096092] LNet: Removed LNI 192.168.204.106@tcp [ 8883.097281] LNet: Removed LNI 192.168.204.106@tcp10 [ 8883.098406] LNet: Removed LNI 192.168.204.106@tcp11 [ 8883.099556] LNet: Removed LNI 192.168.204.106@tcp12 [ 8883.100689] LNet: Removed LNI 192.168.204.106@tcp13 [ 8883.101779] LNet: Removed LNI 192.168.204.106@tcp14 [ 8883.102946] LNet: Removed LNI 192.168.204.106@tcp15 [ 8883.104121] LNet: Removed LNI 192.168.204.106@tcp16 [ 8883.105196] LNet: Removed LNI 192.168.204.106@tcp17 [ 8883.106318] LNet: Removed LNI 192.168.204.106@tcp18 [ 8883.107535] LNet: Removed LNI 192.168.204.106@tcp19 [ 8883.108629] LNet: Removed LNI 192.168.204.106@tcp20 [ 8883.109736] LNet: Removed LNI 192.168.204.106@tcp21 [ 8883.110902] LNet: Removed LNI 192.168.204.106@tcp22 [ 8883.112102] LNet: Removed LNI 192.168.204.106@tcp23 [ 8883.113359] LNet: Removed LNI 192.168.204.106@tcp24 [ 8883.114910] LNet: Removed LNI 192.168.204.106@tcp25 [ 8883.117280] LNet: Removed LNI 192.168.204.106@tcp26 [ 8883.119649] LNet: Removed LNI 192.168.204.106@tcp27 [ 8883.121584] LNet: Removed LNI 192.168.204.106@tcp28 [ 8883.122678] LNet: Removed LNI 192.168.204.106@tcp29 [ 8883.123811] LNet: Removed LNI 192.168.204.106@tcp30 [ 8883.125799] LNet: Removed LNI 192.168.204.106@tcp31 [ 8883.126989] LNet: Removed LNI 192.168.204.106@tcp32 [ 8883.128151] LNet: Removed LNI 192.168.204.106@tcp33 [ 8883.129382] LNet: Removed LNI 192.168.204.106@tcp34 [ 8883.130557] LNet: Removed LNI 192.168.204.106@tcp35 [ 8883.131681] LNet: Removed LNI 192.168.204.106@tcp36 [ 8883.132782] LNet: Removed LNI 192.168.204.106@tcp37 [ 8883.134022] LNet: Removed LNI 192.168.204.106@tcp38 [ 8883.135260] LNet: Removed LNI 192.168.204.106@tcp39 [ 8883.136465] LNet: Removed LNI 192.168.204.106@tcp40 [ 8883.137591] LNet: Removed LNI 192.168.204.106@tcp41 [ 8883.138826] LNet: Removed LNI 192.168.204.106@tcp42 [ 8883.139966] LNet: Removed LNI 192.168.204.106@tcp43 [ 8883.141027] LNet: Removed LNI 192.168.204.106@tcp44 [ 8883.142336] LNet: Removed LNI 192.168.204.106@tcp45 [ 8883.143517] LNet: Removed LNI 192.168.204.106@tcp46 [ 8883.144638] LNet: Removed LNI 192.168.204.106@tcp47 [ 8883.145696] LNet: Removed LNI 192.168.204.106@tcp48 [ 8883.146874] LNet: Removed LNI 192.168.204.106@tcp49 [ 8883.153337] LNet: Removed LNI 192.168.204.106@tcp50 [ 8883.427172] Key type .llcrypt unregistered [ 8883.428302] Key type ._llcrypt unregistered [ 8890.171927] Key type ._llcrypt registered [ 8890.173096] Key type .llcrypt registered [ 8890.211659] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_hostid [ 8896.594535] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 8896.986239] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8896.999221] alg: No test for adler32 (adler32-zlib) [ 8897.858502] Lustre: Lustre: Build Version: 2.17.52_93_g6cdb58a [ 8897.946427] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 8899.528153] Key type lgssc registered [ 8899.900351] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8911.745128] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8911.752622] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8912.852721] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8912.863303] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8912.905634] Lustre: lustre-MDT0000: new disk, initializing [ 8912.930334] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8912.937523] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8914.258536] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 8918.515444] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8918.562876] Lustre: 474050:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8918.595629] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8918.598754] Lustre: Skipped 1 previous similar message [ 8918.634106] Lustre: lustre-MDT0001: new disk, initializing [ 8918.665636] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8918.677487] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8918.681272] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8920.041992] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 8923.196809] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8924.426720] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8927.007544] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8927.091351] Lustre: *** cfs_fail_loc=90f, val=0*** [ 8927.121151] Lustre: lustre-OST0000: new disk, initializing [ 8927.123056] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8927.149944] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8929.153338] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 8932.349934] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8932.353070] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8932.395651] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8932.402606] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8937.441869] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8937.443548] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8937.444385] LustreError: Skipped 1 previous similar message [ 8937.449067] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8937.453204] Lustre: Skipped 1 previous similar message [ 8940.826069] Lustre: server umount lustre-OST0000 complete [ 8942.561789] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8942.562423] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8942.567630] Lustre: Skipped 2 previous similar messages [ 8947.681684] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8947.684672] Lustre: Skipped 2 previous similar messages [ 8948.318941] Lustre: server umount lustre-MDT0000 complete [ 8949.626064] LustreError: 475839:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777505688 with bad export cookie 4274962820291113845 [ 8949.629446] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8949.629533] LustreError: 475839:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8949.747665] Lustre: server umount lustre-MDT0001 complete [ 8953.589198] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 8954.682346] Key type lgssc unregistered [ 8954.807323] LNet: 477610:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8954.809779] LNetError: 477610:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8954.817323] LNet: Removed LNI 192.168.204.106@tcp [ 8955.114031] Key type .llcrypt unregistered [ 8955.115111] Key type ._llcrypt unregistered [ 8962.630854] Key type ._llcrypt registered [ 8962.632606] Key type .llcrypt registered [ 8962.671163] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 8963.139850] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8963.155591] alg: No test for adler32 (adler32-zlib) [ 8964.018522] Lustre: Lustre: Build Version: 2.17.52_93_g6cdb58a [ 8964.100291] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 8965.688149] Key type lgssc registered [ 8966.044132] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8968.706616] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8968.713934] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8969.867238] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8971.132767] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 8974.097239] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8974.242753] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8975.637507] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 8977.512811] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8978.748927] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8980.996689] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8981.104351] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8982.930788] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 8984.737324] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8998.906601] loop0: detected capacity change from 0 to 204800000 [ 8998.982811] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8998.986625] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8998.992286] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9001.060472] loop0: detected capacity change from 0 to 204800000 [ 9001.181215] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9001.186229] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9001.190441] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9002.964648] loop0: detected capacity change from 0 to 204800000 [ 9004.914151] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9004.960380] Lustre: fs15246-OST0000: new disk, initializing [ 9004.961973] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [ 9004.980436] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9006.843541] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 9010.558473] loop1: detected capacity change from 0 to 204800000 [ 9012.427878] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9012.464937] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [ 9012.489194] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [ 9012.541218] Lustre: fs15246-MDT0000: new disk, initializing [ 9012.606277] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9012.616118] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9012.664589] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 9013.728950] Lustre: 483216:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@00000000143a3fc3 x1863849819011328/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1777505768 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 9014.010093] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 9022.816278] LustreError: 483216:0:(fid_request.c:213:seq_client_alloc_seq()) cli-cli-fs15246-OST0000-osc-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 9022.819351] LustreError: 483216:0:(fid_request.c:255:seq_client_get_seq()) cli-cli-fs15246-OST0000-osc-MDT0000: Can't allocate new sequence: rc = -5 [ 9022.822114] LustreError: 483216:0:(osp_precreate.c:460:osp_precreate_rollover_new_seq()) fs15246-OST0000-osc-MDT0000: alloc fid error: rc = -5 [ 9026.022688] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9026.029067] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [ 9027.730657] Lustre: server umount fs15246-MDT0000 complete [ 9030.770708] Lustre: server umount fs15246-OST0000 complete [ 9035.745594] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9035.748697] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9035.753145] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9040.341355] Lustre: server umount lustre-OST0000 complete [ 9045.984955] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9045.987098] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9045.990373] Lustre: Skipped 1 previous similar message [ 9045.992147] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9045.993767] Lustre: Skipped 2 previous similar messages [ 9047.766939] Lustre: server umount lustre-MDT0000 complete [ 9049.055408] LustreError: 478863:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777505787 with bad export cookie 4009844790675059840 [ 9049.058932] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9049.060276] LustreError: 478863:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9049.185417] Lustre: server umount lustre-MDT0001 complete [ 9052.777390] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 9053.825525] Key type lgssc unregistered [ 9053.956452] LNet: 486275:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9053.958855] LNetError: 486275:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9053.965307] LNet: Removed LNI 192.168.204.106@tcp [ 9054.273579] Key type .llcrypt unregistered [ 9054.274725] Key type ._llcrypt unregistered [ 9061.385757] Key type ._llcrypt registered [ 9061.386848] Key type .llcrypt registered [ 9061.424365] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 9061.876215] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9061.885602] alg: No test for adler32 (adler32-zlib) [ 9062.741607] Lustre: Lustre: Build Version: 2.17.52_93_g6cdb58a [ 9062.832703] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 9064.416204] Key type lgssc registered [ 9064.837900] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9077.412231] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_hostid [ 9080.874402] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 9093.106055] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9093.111542] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9094.206225] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9094.217642] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9094.252783] Lustre: lustre-MDT0000: new disk, initializing [ 9094.276725] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9094.283787] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9095.637041] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 9099.900830] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9099.951944] Lustre: 490666:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9099.979105] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9099.981073] Lustre: Skipped 1 previous similar message [ 9100.023347] Lustre: lustre-MDT0001: new disk, initializing [ 9100.059359] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9100.072430] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9100.076738] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9101.374943] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 9104.558561] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9105.766304] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9108.053508] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9108.149479] Lustre: lustre-OST0000: new disk, initializing [ 9108.151219] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9108.178863] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9108.792945] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9108.795771] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9108.827527] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9110.036294] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 9113.095843] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9114.410205] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9114.479108] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9115.750546] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9115.818417] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9119.201761] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9119.203602] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9119.204483] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9119.205604] LustreError: Skipped 1 previous similar message [ 9119.210696] Lustre: Skipped 1 previous similar message [ 9122.579928] Lustre: server umount lustre-OST0000 complete [ 9124.321789] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9124.322299] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9124.325662] Lustre: Skipped 1 previous similar message [ 9124.330121] Lustre: Skipped 2 previous similar messages [ 9129.441065] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9129.443608] Lustre: Skipped 1 previous similar message [ 9134.560962] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9134.563937] Lustre: Skipped 1 previous similar message [ 9138.144147] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9138.204699] Lustre: server umount lustre-MDT0000 complete [ 9139.483189] LustreError: 490659:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777505878 with bad export cookie 18337128492644274192 [ 9139.486169] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9139.487405] LustreError: 490659:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9139.613450] Lustre: server umount lustre-MDT0001 complete [ 9147.568918] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_hostid [ 9151.034691] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 9161.515989] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 8895 sec ========== 19:38:19 (1777505899) [ 9162.074732] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 19:38:19 (1777505899) === [ 9163.255426] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 19:38:21 (1777505901) === [ 9176.468329] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 9177.449616] Key type lgssc unregistered [ 9177.578292] LNet: 498843:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9177.580698] LNetError: 498843:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9177.587360] LNet: Removed LNI 192.168.204.106@tcp [ 9177.877058] Key type .llcrypt unregistered [ 9177.878218] Key type ._llcrypt unregistered