[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 498610771 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001018] APIC: Switch to symmetric I/O mode setup [ 0.003348] x2apic enabled [ 0.004013] Switched APIC routing to physical x2apic. [ 0.006014] kvm-guest: setup PV IPIs [ 0.009321] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.010013] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.011033] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.012022] pid_max: default: 32768 minimum: 301 [ 0.014149] LSM: Security Framework initializing [ 0.015092] Yama: becoming mindful. [ 0.017075] SELinux: Initializing. [ 0.018110] *** VALIDATE selinux *** [ 0.026446] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.030795] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.031159] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032138] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.033154] *** VALIDATE tmpfs *** [ 0.034549] *** VALIDATE proc *** [ 0.035335] *** VALIDATE cgroup *** [ 0.036020] *** VALIDATE cgroup2 *** [ 0.037325] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.039162] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.040012] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.041042] Spectre V2 : User space: Vulnerable [ 0.042014] Speculative Store Bypass: Vulnerable [ 0.045306] debug: unmapping init [mem 0xffffffff8d659000-0xffffffff8d660fff] [ 0.048196] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.049780] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.050034] ... version: 2 [ 0.051019] ... bit width: 48 [ 0.052018] ... generic registers: 4 [ 0.053015] ... value mask: 0000ffffffffffff [ 0.054021] ... max period: 00007fffffffffff [ 0.055022] ... fixed-purpose events: 3 [ 0.056014] ... event mask: 000000070000000f [ 0.057356] rcu: Hierarchical SRCU implementation. [ 0.059753] smp: Bringing up secondary CPUs ... [ 0.060688] x86: Booting SMP configuration: [ 0.061038] .... node #0, CPUs: #1 #2 #3 [ 0.065026] smp: Brought up 1 node, 4 CPUs [ 0.067029] smpboot: Max logical packages: 1 [ 0.068032] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.251020] node 0 deferred pages initialised in 180ms [ 0.255019] devtmpfs: initialized [ 0.256291] x86/mm: Memory block size: 128MB [ 0.258987] gcov: version magic: 0x41383552 [ 0.260366] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.261113] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.262352] pinctrl core: initialized pinctrl subsystem [ 0.263269] [ 0.263744] ************************************************************* [ 0.264023] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.265017] ** ** [ 0.266018] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.267034] ** ** [ 0.268018] ** This means that this kernel is built to expose internal ** [ 0.269022] ** IOMMU data structures, which may compromise security on ** [ 0.270019] ** your system. ** [ 0.271021] ** ** [ 0.272020] ** If you see this message and you are not debugging the ** [ 0.273020] ** kernel, report this immediately to your vendor! ** [ 0.274016] ** ** [ 0.275018] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.276023] ************************************************************* [ 0.277879] NET: Registered protocol family 16 [ 0.278566] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.279081] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.280227] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.281844] cpuidle: using governor menu [ 0.283980] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.286760] PCI: Using configuration type 1 for base access [ 0.289219] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.299130] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.300026] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.302089] cryptd: max_cpu_qlen set to 1000 [ 0.304330] ACPI: Added _OSI(Module Device) [ 0.305017] ACPI: Added _OSI(Processor Device) [ 0.306020] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.307016] ACPI: Added _OSI(Processor Aggregator Device) [ 0.311687] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.315289] ACPI: Interpreter enabled [ 0.316073] ACPI: PM: (supports S0 S3 S4 S5) [ 0.317013] ACPI: Using IOAPIC for interrupt routing [ 0.318170] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.319456] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.329858] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.330060] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.331027] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.332166] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.334592] acpiphp: Slot [2] registered [ 0.335364] acpiphp: Slot [5] registered [ 0.336201] acpiphp: Slot [6] registered [ 0.337163] acpiphp: Slot [7] registered [ 0.338221] acpiphp: Slot [8] registered [ 0.339154] acpiphp: Slot [9] registered [ 0.340167] acpiphp: Slot [10] registered [ 0.341155] acpiphp: Slot [3] registered [ 0.342206] acpiphp: Slot [4] registered [ 0.343107] acpiphp: Slot [11] registered [ 0.344198] acpiphp: Slot [12] registered [ 0.345124] acpiphp: Slot [13] registered [ 0.346165] acpiphp: Slot [14] registered [ 0.347095] acpiphp: Slot [15] registered [ 0.349210] acpiphp: Slot [16] registered [ 0.350161] acpiphp: Slot [17] registered [ 0.351092] acpiphp: Slot [18] registered [ 0.352098] acpiphp: Slot [19] registered [ 0.354101] acpiphp: Slot [20] registered [ 0.355084] acpiphp: Slot [21] registered [ 0.356158] acpiphp: Slot [22] registered [ 0.357081] acpiphp: Slot [23] registered [ 0.358346] acpiphp: Slot [24] registered [ 0.359111] acpiphp: Slot [25] registered [ 0.361112] acpiphp: Slot [26] registered [ 0.362081] acpiphp: Slot [27] registered [ 0.363091] acpiphp: Slot [28] registered [ 0.364084] acpiphp: Slot [29] registered [ 0.365101] acpiphp: Slot [30] registered [ 0.366077] acpiphp: Slot [31] registered [ 0.367081] PCI host bridge to bus 0000:00 [ 0.369019] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.370021] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.372024] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.374024] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.376024] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.378032] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.380166] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.383060] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.388590] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.400825] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.404882] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.407026] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.410029] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.414029] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.416642] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.420019] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.423069] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.427338] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.433021] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.446893] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.452018] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.458215] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.466024] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.474024] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.495026] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.508689] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.517018] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.523020] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.536025] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.546909] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.552022] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.561018] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.574025] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.582731] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.590020] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.597022] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.614022] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.625648] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.631022] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.641017] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.654038] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.664062] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.671018] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.683021] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.712022] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.727456] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.730365] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.732366] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.735375] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.737244] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.743160] iommu: Default domain type: Passthrough [ 0.745645] SCSI subsystem initialized [ 0.747207] ACPI: bus type USB registered [ 0.749104] usbcore: registered new interface driver usbfs [ 0.751076] usbcore: registered new interface driver hub [ 0.752090] usbcore: registered new device driver usb [ 0.754170] pps_core: LinuxPPS API ver. 1 registered [ 0.756018] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.759063] PTP clock support registered [ 0.761132] EDAC MC: Ver: 3.0.0 [ 0.763163] PCI: Using ACPI for IRQ routing [ 0.765953] NetLabel: Initializing [ 0.767016] NetLabel: domain hash size = 128 [ 0.769015] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.770105] NetLabel: unlabeled traffic allowed by default [ 0.773060] vgaarb: loaded [ 0.774310] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.776019] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.783407] clocksource: Switched to clocksource kvm-clock [ 0.895150] VFS: Disk quotas dquot_6.6.0 [ 0.897027] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.900067] *** VALIDATE ramfs *** [ 0.901491] *** VALIDATE hugetlbfs *** [ 0.903228] pnp: PnP ACPI init [ 0.906050] pnp: PnP ACPI: found 6 devices [ 0.925686] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.929311] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.931517] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.933673] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.936503] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.939168] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.942467] NET: Registered protocol family 2 [ 0.945293] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.950462] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.954482] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.960532] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.964314] TCP: Hash tables configured (established 65536 bind 65536) [ 0.967726] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.971072] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.974084] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.977143] NET: Registered protocol family 1 [ 0.979862] RPC: Registered named UNIX socket transport module. [ 0.982098] RPC: Registered udp transport module. [ 0.984165] RPC: Registered tcp transport module. [ 0.986095] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.988825] NET: Registered protocol family 44 [ 0.990786] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.992697] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.995173] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.998022] PCI: CLS 0 bytes, default 64 [ 1.000051] Unpacking initramfs... [ 2.448586] debug: unmapping init [mem 0xffff98c77cc54000-0xffff98c77ffbffff] [ 2.452885] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.455661] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.458975] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.969706] Initialise system trusted keyrings [ 2.971524] Key type blacklist registered [ 2.973438] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.983209] zbud: loaded [ 2.986800] *** VALIDATE nfs *** [ 2.988249] *** VALIDATE nfs4 *** [ 2.990056] pstore: using deflate compression [ 2.995067] Platform Keyring initialized [ 3.102396] NET: Registered protocol family 38 [ 3.103667] Key type asymmetric registered [ 3.104821] Asymmetric key parser 'x509' registered [ 3.106652] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.110093] io scheduler mq-deadline registered [ 3.111431] io scheduler kyber registered [ 3.112579] io scheduler bfq registered [ 3.114412] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.116759] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.118734] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.120861] ACPI: Power Button [PWRF] [ 3.126435] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.132725] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.149057] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.161969] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.183541] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.212234] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.239923] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.244257] Non-volatile memory driver v1.3 [ 3.245908] Linux agpgart interface v0.103 [ 3.280683] virtio_blk virtio1: [vda] 145960 512-byte logical blocks (74.7 MB/71.3 MiB) [ 3.283839] vda: detected capacity change from 0 to 74731520 [ 3.300866] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.304709] vdb: detected capacity change from 0 to 1073741824 [ 3.326085] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.329348] vdc: detected capacity change from 0 to 2621440000 [ 3.343945] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.346383] vdd: detected capacity change from 0 to 2621440000 [ 3.361509] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.364841] vde: detected capacity change from 0 to 4294967296 [ 3.380419] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.383641] vdf: detected capacity change from 0 to 4294967296 [ 3.393450] libphy: Fixed MDIO Bus: probed [ 3.414138] usbcore: registered new interface driver usbserial_generic [ 3.416809] usbserial: USB Serial support registered for generic [ 3.419271] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.423977] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.425624] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.427667] mousedev: PS/2 mouse device common for all mice [ 3.432504] rtc_cmos 00:05: RTC can wake from S4 [ 3.432878] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.438891] rtc_cmos 00:05: registered as rtc0 [ 3.441217] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.443181] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.445153] intel_pstate: CPU model not supported [ 3.451076] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.455641] hid: raw HID events driver (C) Jiri Kosina [ 3.457996] usbcore: registered new interface driver usbhid [ 3.459867] usbhid: USB HID core driver [ 3.461316] drop_monitor: Initializing network drop monitor service [ 3.464190] Initializing XFRM netlink socket [ 3.466519] NET: Registered protocol family 10 [ 3.470063] Segment Routing with IPv6 [ 3.472505] NET: Registered protocol family 17 [ 3.475490] mpls_gso: MPLS GSO support [ 3.482775] RAS: Correctable Errors collector initialized. [ 3.485415] AVX version of gcm_enc/dec engaged. [ 3.487641] AES CTR mode by8 optimization enabled [ 3.574610] sched_clock: Marking stable (3574585858, 0)->(4639651746, -1065065888) [ 3.581664] registered taskstats version 1 [ 3.584259] Loading compiled-in X.509 certificates [ 3.587827] zswap: loaded using pool lzo/zbud [ 3.615101] Key type big_key registered [ 3.627590] Key type encrypted registered [ 3.629474] ima: No TPM chip found, activating TPM-bypass! [ 3.631862] ima: Allocated hash algorithm: sha1 [ 3.633881] ima: No architecture policies found [ 3.635917] evm: Initialising EVM extended attributes: [ 3.638212] evm: security.selinux [ 3.639646] evm: security.ima [ 3.641011] evm: security.capability [ 3.642610] evm: HMAC attrs: 0x1 [ 3.645526] rtc_cmos 00:05: setting system clock to 2026-09-08 21:32:07 UTC (1788903127) [ 3.652115] debug: unmapping init [mem 0xffffffff8e603000-0xffffffff8e7fffff] [ 3.655447] debug: unmapping init [mem 0xffffffff8d382000-0xffffffff8d658fff] [ 3.668105] Write protecting the kernel read-only data: 28672k [ 3.671698] debug: unmapping init [mem 0xffffffff8ba03000-0xffffffff8bbfffff] [ 3.675114] debug: unmapping init [mem 0xffffffff8c314000-0xffffffff8c3fffff] [ 3.710159] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.719857] systemd[1]: Detected virtualization kvm. [ 3.721932] systemd[1]: Detected architecture x86-64. [ 3.725574] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.757321] systemd[1]: No hostname configured. [ 3.759212] systemd[1]: Set hostname to . [ 3.761220] random: systemd: uninitialized urandom read (16 bytes read) [ 3.764091] systemd[1]: Initializing machine ID from random generator. [ 3.795527] random: ln: uninitialized urandom read (6 bytes read) [ 3.916187] random: systemd: uninitialized urandom read (16 bytes read) [ 3.919746] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.925791] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.935680] systemd[1]: Starting Setup Virtual Console... Starting Setup Virtual Console... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Swap. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Timers. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.587600] device-mapper: uevent: version 1.0.3 [ 4.590765] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.361570] random: fast init done [ 5.414218] virtio_net virtio0 ens2: renamed from eth0 [ 5.519571] scsi host0: ata_piix [ 5.558552] scsi host1: ata_piix [ 5.566450] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.569099] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.234859] dracut-initqueue[588]: RTNETLINK answers: File exists [ 10.147680] random: crng init done [ 10.149896] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.657793] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.821756] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.077862] SELinux: Disabled at runtime. [ 12.140707] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.150433] systemd[1]: Detected virtualization kvm. [ 12.152253] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.720157] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.723343] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.728769] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.733022] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.738218] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.747341] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.751444] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. Mounting POSIX Message Queue File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Reached target rpc_pipefs.target. [ OK ] Stopped target Switch Root. Starting Create list of required st…ce nodes for the current kernel... Activating swap /dev/disk/by-label/SWAP... Mounting Huge Pages File System... [ OK ] Created slice system-getty.slice. [ OK ] Stopped target Initrd Root File System. Starting Remount Root and Kernel File Systems... [ 12.833242] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on Process Core Dump Socket. [ OK ] Stopped target Initrd File Systems. Starting Apply Kernel Variables... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Listening on initctl Compatibility Named Pipe. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting Kernel Debug File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt[ 13.187279] squashfs: version 4.0 (2009/01/31) Phillip Lougher . [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.526539] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.536903] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.663232] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.675282] EDAC sbridge: Ver: 1.1.2 [ 15.561567] Key type dns_resolver registered [ 15.889561] NFS: Registering the id_resolver key type [ 15.891757] Key type id_resolver registered [ 15.893620] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Mark the need to relabel after reboot... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Restore /run/initramfs on shutdown... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started OpenSSH server daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg426-server login: [ 41.841355] libcfs: loading out-of-tree module taints kernel. [ 41.864594] Key type ._llcrypt registered [ 41.866205] Key type .llcrypt registered [ 41.923598] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [ 51.344756] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 52.823405] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 52.852927] alg: No test for adler32 (adler32-zlib) [ 54.302843] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 55.577745] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 57.424052] Key type lgssc registered [ 59.030923] Lustre: Echo OBD driver; http://www.lustre.org/ [ 82.481494] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 108.775953] hrtimer: interrupt took 26952121 ns [ 132.758075] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 146.505505] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 146.540199] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 147.869934] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 147.922742] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 148.019605] Lustre: lustre-MDT0000: new disk, initializing [ 148.117570] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 148.134314] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 152.258798] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 166.154475] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 166.273805] Lustre: 6494:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 166.363848] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 166.377181] Lustre: Skipped 1 previous similar message [ 166.567553] Lustre: lustre-MDT0001: new disk, initializing [ 166.574717] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 166.663079] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 166.689788] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 166.709573] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 171.937793] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 177.006888] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 189.192686] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 189.669642] Lustre: lustre-OST0000: new disk, initializing [ 189.678603] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 189.690384] Lustre: 8431:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 189.794424] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 192.199840] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 192.218958] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 192.364859] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 197.666569] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 216.858881] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 217.065429] Lustre: lustre-OST0001: new disk, initializing [ 217.074920] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 217.085067] Lustre: 9505:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 217.194393] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 224.311646] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 224.325083] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 224.525608] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 225.469787] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 239.134223] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 251.154200] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 260.540988] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing check_logdir /tmp/testlogs/ [ 267.095222] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing yml_node [ 272.702832] Lustre: DEBUG MARKER: Client: 2.17.55.26 [ 276.780341] Lustre: DEBUG MARKER: MDS: 2.17.55.26 [ 279.938811] Lustre: DEBUG MARKER: OSS: 2.17.55.26 [ 282.209221] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Sep 8 17:36:43 EDT 2026 [ 299.184490] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 300.616446] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 316.896338] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 316.896758] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 316.914635] Lustre: Skipped 1 previous similar message [ 321.840072] Lustre: server umount lustre-MDT0000 complete [ 324.843633] LustreError: 6485:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788903448 with bad export cookie 1752257781896379150 [ 324.848951] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 324.856191] LustreError: 6485:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 327.138924] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 327.140976] LustreError: 6502:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 327.142172] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 327.142178] Lustre: Skipped 3 previous similar messages [ 327.164146] Lustre: Skipped 3 previous similar messages [ 327.180441] LustreError: 6502:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 329.185891] LustreError: 8556:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 329.188635] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 329.207889] Lustre: Skipped 2 previous similar messages [ 331.247608] Lustre: server umount lustre-MDT0001 complete [ 334.972058] Lustre: server umount lustre-OST0000 complete [ 338.022548] Lustre: server umount lustre-OST0001 complete [ 342.972054] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [ 349.039961] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 368.090957] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 368.225933] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 368.249828] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 368.297602] Lustre: lustre-MDT0000: new disk, initializing [ 368.351366] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 368.364159] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 370.617076] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 376.670535] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 376.744768] Lustre: 15056:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 376.787481] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 376.791506] Lustre: Skipped 1 previous similar message [ 376.843346] Lustre: lustre-MDT0001: new disk, initializing [ 376.915737] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 376.975672] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 376.985658] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 379.355986] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 384.584959] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 387.640905] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 391.764636] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 391.940606] Lustre: 17025:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 393.430438] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 393.436573] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 393.510159] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 396.048616] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 401.499850] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 403.694243] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 403.825383] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 406.002597] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 406.127827] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 409.058981] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 409.064566] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 409.066836] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 409.078615] Lustre: Skipped 1 previous similar message [ 409.086218] Lustre: Skipped 1 previous similar message [ 419.298821] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 419.304216] Lustre: Skipped 3 previous similar messages [ 421.343711] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 421.436876] Lustre: server umount lustre-OST0000 complete [ 424.416788] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 424.427774] Lustre: Skipped 1 previous similar message [ 429.286688] Lustre: server umount lustre-MDT0000 complete [ 429.536671] LustreError: 15544:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 429.546099] LustreError: 15544:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 430.925813] LustreError: 15049:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788903554 with bad export cookie 1752257781896380683 [ 430.928485] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 430.933054] LustreError: 15049:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 431.085428] Lustre: server umount lustre-MDT0001 complete [ 432.934330] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 17:39:15 (1788903555) [ 434.104921] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 434.901895] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 17:39:17 (1788903557) [ 438.154256] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 438.391704] LustreError: 19431:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 438.426779] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 438.430969] Lustre: Skipped 1 previous similar message [ 440.341764] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 443.876905] LustreError: 19432:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 444.682642] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 447.077540] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 450.747884] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 453.500560] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 456.820792] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 459.583128] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 462.933222] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 479.713174] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 479.717977] LustreError: Skipped 1 previous similar message [ 479.720756] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 479.729371] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 479.733619] Lustre: Skipped 3 previous similar messages [ 481.890930] Lustre: server umount lustre-OST0000 complete [ 489.780605] Lustre: server umount lustre-MDT0000 complete [ 489.952507] LustreError: 19432:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 489.961322] LustreError: 19432:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 491.433485] LustreError: 19414:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788903615 with bad export cookie 1752257781896381565 [ 491.436108] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 491.446608] LustreError: 19414:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 491.605285] Lustre: server umount lustre-MDT0001 complete [ 496.252331] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 497.622372] Key type lgssc unregistered [ 497.791575] LNet: 22980:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 497.795920] LNetError: 22980:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 497.804405] LNet: Removed LNI 192.168.204.126@tcp [ 498.300144] Key type .llcrypt unregistered [ 498.302600] Key type ._llcrypt unregistered [ 509.107944] Key type ._llcrypt registered [ 509.109585] Key type .llcrypt registered [ 509.160213] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 509.602676] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 509.624025] alg: No test for adler32 (adler32-zlib) [ 510.519287] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 510.635095] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 512.231189] Key type lgssc registered [ 512.736754] Lustre: Echo OBD driver; http://www.lustre.org/ [ 516.446390] loop: module loaded [ 516.473436] loop0: detected capacity change from 0 to 204800000 [ 516.563555] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 516.569308] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 516.713629] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 519.801271] loop0: detected capacity change from 0 to 204800000 [ 519.874137] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 519.880376] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 519.886832] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 522.651260] loop0: detected capacity change from 0 to 204800000 [ 525.423324] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 525.432468] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 526.551686] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 526.586992] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 526.631170] Lustre: lustre-MDT0000: new disk, initializing [ 526.670349] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 526.690203] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 528.887542] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 533.858255] loop1: detected capacity change from 0 to 204800000 [ 536.561235] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 536.680345] Lustre: lustre-OST07c6: new disk, initializing [ 536.682972] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 536.685771] Lustre: Skipped 1 previous similar message [ 536.688970] Lustre: 26144:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 536.730847] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 539.737784] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 542.740816] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 542.746356] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 542.764541] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 547.980577] Lustre: 25128:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 547.989760] Lustre: 25128:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 547.997274] Lustre: 25128:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 548.000734] Lustre: 25128:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 548.004568] Lustre: 25128:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 548.009677] Lustre: 25128:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 564.703267] Lustre: 25157:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 564.707133] Lustre: 25157:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 564.712728] Lustre: 25157:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 564.717368] Lustre: 25157:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 564.723355] Lustre: 25157:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 564.728444] Lustre: 25157:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 568.573958] Lustre: 25129:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 568.577508] Lustre: 25129:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 568.581279] Lustre: 25129:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 568.584895] Lustre: 25129:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 568.588432] Lustre: 25129:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 568.592242] Lustre: 25129:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 568.595990] Lustre: 25129:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 568.599892] Lustre: 25129:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 568.603718] Lustre: 25129:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 568.608476] Lustre: 25129:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 568.613247] Lustre: 25129:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 568.616394] Lustre: 25129:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 571.706510] Lustre: 25157:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 571.710237] Lustre: 25157:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 571.712845] Lustre: 25157:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 571.715389] Lustre: 25157:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 571.718889] Lustre: 25157:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 571.722552] Lustre: 25157:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 571.725801] Lustre: 25157:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 571.729086] Lustre: 25157:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 571.731995] Lustre: 25157:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 571.734967] Lustre: 25157:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 571.738047] Lustre: 25157:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 571.741936] Lustre: 25157:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 578.531416] Lustre: 25127:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 578.536071] Lustre: 25127:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 3 previous similar messages [ 578.540538] Lustre: 25127:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 578.544270] Lustre: 25127:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 578.548669] Lustre: 25127:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 578.552331] Lustre: 25127:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 578.555876] Lustre: 25127:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 578.559095] Lustre: 25127:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 578.562026] Lustre: 25127:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 578.564661] Lustre: 25127:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 578.567224] Lustre: 25127:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 578.570394] Lustre: 25127:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 588.942106] Lustre: 25128:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 588.948542] Lustre: 25128:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 5 previous similar messages [ 588.955306] Lustre: 25128:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 588.958613] Lustre: 25128:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 588.962311] Lustre: 25128:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 588.965573] Lustre: 25128:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 588.968519] Lustre: 25128:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 588.971701] Lustre: 25128:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 588.974418] Lustre: 25128:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 588.977518] Lustre: 25128:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 588.981067] Lustre: 25128:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 588.984362] Lustre: 25128:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 634.850039] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 634.850441] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 634.860115] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 635.986714] Lustre: server umount lustre-OST07c6 complete [ 639.614884] Lustre: server umount lustre-MDT0000 complete [ 646.004382] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 647.199232] Key type lgssc unregistered [ 647.351643] LNet: 28720:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 647.356280] LNetError: 28720:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 647.370370] LNet: Removed LNI 192.168.204.126@tcp [ 647.778200] Key type .llcrypt unregistered [ 647.780157] Key type ._llcrypt unregistered [ 655.824889] Key type ._llcrypt registered [ 655.826048] Key type .llcrypt registered [ 655.862668] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 656.369076] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 656.378593] alg: No test for adler32 (adler32-zlib) [ 657.233354] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 657.319634] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 658.911152] Key type lgssc registered [ 659.303383] Lustre: Echo OBD driver; http://www.lustre.org/ [ 662.286613] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 662.297162] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 663.503026] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 665.356270] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 668.903502] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 669.109453] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 670.804425] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 673.762105] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 676.080562] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 678.658568] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 678.793553] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 680.996247] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 683.727304] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 689.062287] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 699.359946] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 699.364488] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 699.369904] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 703.456685] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 703.457644] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 703.459649] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 704.903563] Lustre: server umount lustre-OST0000 complete [ 708.576460] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 708.577258] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 708.580548] Lustre: Skipped 1 previous similar message [ 708.582642] Lustre: Skipped 1 previous similar message [ 712.277917] Lustre: server umount lustre-MDT0000 complete [ 713.525700] LustreError: 32770:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788903837 with bad export cookie 18176863228958584777 [ 713.530216] LustreError: 32770:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 713.530269] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 713.642646] Lustre: server umount lustre-MDT0001 complete [ 717.365263] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 718.378996] Key type lgssc unregistered [ 718.513458] LNet: 33722:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 718.517184] LNetError: 33722:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 718.524306] LNet: Removed LNI 192.168.204.126@tcp [ 718.833552] Key type .llcrypt unregistered [ 718.834671] Key type ._llcrypt unregistered [ 725.759463] Key type ._llcrypt registered [ 725.761103] Key type .llcrypt registered [ 725.800245] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 726.227703] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 726.236168] alg: No test for adler32 (adler32-zlib) [ 727.078634] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 727.154092] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 728.727157] Key type lgssc registered [ 729.074423] Lustre: Echo OBD driver; http://www.lustre.org/ [ 731.519659] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 731.525348] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 732.672292] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 733.917290] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 736.791034] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 736.924424] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 738.274791] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 740.682672] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 742.464499] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 744.602595] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 744.704454] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 746.477929] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 748.712577] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 759.082076] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 764.386503] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 764.386601] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 764.391586] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 764.396092] Lustre: Skipped 1 previous similar message [ 765.068442] Lustre: server umount lustre-OST0000 complete [ 767.967954] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 767.971579] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 767.975586] Lustre: Skipped 1 previous similar message [ 767.977294] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 768.463697] Lustre: server umount lustre-MDT0000 complete [ 769.504524] LustreError: 34967:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 769.505247] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 769.511423] LustreError: 34967:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 769.698462] LustreError: 37465:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788903893 with bad export cookie 1577688343494992525 [ 769.701719] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 769.702508] LustreError: 37465:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 769.817629] Lustre: server umount lustre-MDT0001 complete [ 773.485353] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 774.479643] Key type lgssc unregistered [ 774.607498] LNet: 38515:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 774.611111] LNetError: 38515:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 774.618315] LNet: Removed LNI 192.168.204.126@tcp [ 774.914044] Key type .llcrypt unregistered [ 774.915088] Key type ._llcrypt unregistered [ 783.914781] Key type ._llcrypt registered [ 783.915712] Key type .llcrypt registered [ 783.958321] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 784.344901] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 784.362241] alg: No test for adler32 (adler32-zlib) [ 785.204188] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 785.285957] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 786.863142] Key type lgssc registered [ 787.244626] Lustre: Echo OBD driver; http://www.lustre.org/ [ 789.765863] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 789.772914] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 790.918983] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 792.220106] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 795.065255] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 795.201926] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 796.543440] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 798.905652] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 800.718726] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 803.089152] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 803.206237] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 805.146449] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 807.520315] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 816.549374] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 820.704542] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 820.708725] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 820.714318] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 821.728527] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 821.729626] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 821.732172] Lustre: Skipped 1 previous similar message [ 821.737218] Lustre: Skipped 1 previous similar message [ 822.675840] Lustre: server umount lustre-MDT0000 complete [ 823.324484] LustreError: 39934:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 823.330985] LustreError: 39934:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 823.875166] LustreError: 39890:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788903947 with bad export cookie 6607921337806271135 [ 823.877598] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 823.880660] LustreError: 39890:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 826.015692] Lustre: server umount lustre-MDT0001 complete [ 836.891244] LustreError: 41749:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.26@tcp arrived at 1788903960 with bad export cookie 6607921337806271786 [ 836.895322] LustreError: 41749:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 862.175085] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 862.229739] Lustre: server umount lustre-OST0000 complete [ 867.534135] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 868.462265] Key type lgssc unregistered [ 868.589279] LNet: 43787:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 868.591585] LNetError: 43787:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 868.600241] LNet: Removed LNI 192.168.204.126@tcp [ 868.863476] Key type .llcrypt unregistered [ 868.864455] Key type ._llcrypt unregistered [ 877.333049] Key type ._llcrypt registered [ 877.333928] Key type .llcrypt registered [ 877.364587] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 877.655778] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 877.678051] alg: No test for adler32 (adler32-zlib) [ 878.535040] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 878.619187] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 880.191161] Key type lgssc registered [ 880.526467] Lustre: Echo OBD driver; http://www.lustre.org/ [ 882.937373] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 882.943301] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 884.074339] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 885.292313] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 888.037895] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 888.146539] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 889.357632] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 891.534928] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 893.238485] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 895.306034] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 895.406992] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 897.140670] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 899.403435] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 908.710277] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 912.863927] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 912.866191] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 912.870714] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 913.889659] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 913.892045] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 913.893482] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 913.898115] Lustre: Skipped 1 previous similar message [ 914.970914] Lustre: lustre-OST0000: Not available for connect from 192.168.204.26@tcp (stopping) [ 918.153087] Lustre: server umount lustre-OST0000 complete [ 930.709020] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 930.710664] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 930.712746] Lustre: Skipped 1 previous similar message [ 935.904188] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 935.907305] Lustre: Skipped 2 previous similar messages [ 936.914411] Lustre: server umount lustre-MDT0000 complete [ 938.098240] LustreError: 45161:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788904061 with bad export cookie 1307043374674283080 [ 938.102182] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 938.103042] LustreError: 45161:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 944.236684] Lustre: server umount lustre-MDT0001 complete [ 947.802551] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 948.818927] Key type lgssc unregistered [ 948.942314] LNet: 48907:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 948.944638] LNetError: 48907:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 948.952304] LNet: Removed LNI 192.168.204.126@tcp [ 949.242433] Key type .llcrypt unregistered [ 949.243883] Key type ._llcrypt unregistered [ 958.070754] Key type ._llcrypt registered [ 958.071907] Key type .llcrypt registered [ 958.102960] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 958.398874] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 958.423056] alg: No test for adler32 (adler32-zlib) [ 959.279716] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 959.381229] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 960.975150] Key type lgssc registered [ 961.315794] Lustre: Echo OBD driver; http://www.lustre.org/ [ 963.726653] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 963.732460] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 964.862638] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 966.017945] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 968.653043] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 968.777274] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 970.044450] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 972.334130] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 974.026602] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 976.333789] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 976.448928] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 978.406808] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 980.813181] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 989.794375] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 993.220910] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 994.247493] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1004.984299] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-09-0817h48m47s [ 1005.470212] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1006.617414] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.26@tcp (stopping) [ 1007.635943] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.26@tcp (stopping) [ 1009.632151] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1009.634419] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1009.638315] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1013.780146] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.26@tcp (stopping) [ 1013.782667] Lustre: Skipped 5 previous similar messages [ 1018.836209] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.26@tcp (stopping) [ 1018.838696] Lustre: Skipped 3 previous similar messages [ 1020.383097] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1020.445069] Lustre: server umount lustre-MDT0000 complete [ 1021.509233] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1022.945424] LustreError: 50303:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1022.951109] LustreError: 50303:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 1023.681749] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1023.720852] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1023.818971] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1023.840205] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1024.870830] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1025.090706] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1026.149637] Lustre: DEBUG MARKER: Wait for df (36602) ... [ 1026.659359] Lustre: DEBUG MARKER: done [ 1029.088086] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1029.090387] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1029.093796] Lustre: Skipped 2 previous similar messages [ 1029.096268] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1030.114979] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1030.115275] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1033.683689] Lustre: server umount lustre-OST0000 complete [ 1035.232385] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1035.232603] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1035.236838] Lustre: Skipped 1 previous similar message [ 1041.400395] Lustre: server umount lustre-MDT0000 complete [ 1044.184151] LustreError: 50282:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788904168 with bad export cookie 6905988774101702691 [ 1044.188040] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1044.195805] LustreError: 50282:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1044.436618] Lustre: server umount lustre-MDT0001 complete [ 1052.773127] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 1054.683533] Key type lgssc unregistered [ 1054.872917] LNet: 55236:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1054.878929] LNetError: 55236:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1054.889784] LNet: Removed LNI 192.168.204.126@tcp [ 1055.481683] Key type .llcrypt unregistered [ 1055.484119] Key type ._llcrypt unregistered [ 1081.063366] Key type ._llcrypt registered [ 1081.064865] Key type .llcrypt registered [ 1081.135059] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 1081.637298] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1081.733086] alg: No test for adler32 (adler32-zlib) [ 1082.686900] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 1082.819464] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 1084.431126] Key type lgssc registered [ 1084.899701] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1088.156697] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1088.163950] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1089.278897] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1089.289018] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1089.366571] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1091.481294] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1095.373673] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1095.439620] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1095.466624] Lustre: 57421:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1095.613456] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1099.582455] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1111.459495] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1119.643562] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1128.652666] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1128.925405] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1129.214557] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1137.154268] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1137.848393] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1150.971836] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1161.982310] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-09-0817h51m22s [ 1165.891186] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1177.622854] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1180.506932] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1180.799398] LustreError: 57429:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1201.222798] Lustre: lustre-MDT0000: Client d7b658fa-688f-4ff7-8b2d-01d554831584 (at 192.168.204.26@tcp) reconnecting [ 1220.815138] LustreError: 57429:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1223.580862] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1227.909836] Lustre: DEBUG MARKER: done [ 1240.040174] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1240.052196] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1240.077561] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1240.551809] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1241.704045] Lustre: server umount lustre-OST0000 complete [ 1245.666694] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1245.680448] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1245.697138] Lustre: Skipped 2 previous similar messages [ 1245.712183] Lustre: Skipped 2 previous similar messages [ 1246.765577] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1246.774898] Lustre: Skipped 1 previous similar message [ 1251.189152] Lustre: server umount lustre-MDT0000 complete [ 1253.998981] LustreError: 57414:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788904377 with bad export cookie 13271709311470079380 [ 1254.004843] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1254.011955] LustreError: 57414:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1254.333070] Lustre: server umount lustre-MDT0001 complete [ 1265.995513] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 1268.898728] Key type lgssc unregistered [ 1269.140481] LNet: 61753:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1269.151512] LNetError: 61753:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1269.169687] LNet: Removed LNI 192.168.204.126@tcp [ 1269.927205] Key type .llcrypt unregistered [ 1269.932164] Key type ._llcrypt unregistered [ 1316.157451] Key type ._llcrypt registered [ 1316.160027] Key type .llcrypt registered [ 1316.278427] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 1317.046920] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1317.234562] alg: No test for adler32 (adler32-zlib) [ 1318.296713] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 1318.619741] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 1320.407249] Key type lgssc registered [ 1321.477640] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1324.455851] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1324.477617] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1325.777331] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1325.805278] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1325.980737] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1326.794282] Lustre: Failing over lustre-MDT0000 [ 1327.033831] Lustre: server umount lustre-MDT0000 complete [ 1334.199263] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 17:54:15 (1788904455) [ 1340.270315] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1340.590501] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1345.264284] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1355.083155] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1355.195802] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1355.223199] Lustre: 64554:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1355.520706] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1359.767509] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1369.005663] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1376.565469] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1383.963328] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1384.187879] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1384.394425] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1390.691409] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1393.296679] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1398.985829] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1405.418413] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1414.119586] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1414.122595] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1414.133123] Lustre: Skipped 1 previous similar message [ 1414.148884] Lustre: Skipped 2 previous similar messages [ 1419.237582] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1419.250099] Lustre: Skipped 2 previous similar messages [ 1419.927616] Lustre: server umount lustre-MDT0000 complete [ 1424.523895] LustreError: 66409:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788904548 with bad export cookie 18055036625358528319 [ 1424.547633] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1424.553725] LustreError: 66409:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1425.254793] Lustre: server umount lustre-MDT0001 complete [ 1429.187795] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1435.097807] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1444.092702] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1445.855329] Lustre: 63478:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788904553/real 1788904553] req@00000000bbd2943c x1875802264217216/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788904569 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1445.891048] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1445.911244] Lustre: Skipped 1 previous similar message [ 1450.445212] LustreError: 66402:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1450.457532] LustreError: 66402:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 1450.511853] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1450.616984] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1451.492786] Lustre: 63475:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788904558/real 1788904558] req@0000000095a23efe x1875802264217344/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788904574 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1454.633072] LustreError: 66403:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1454.633833] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1455.128786] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1455.661919] LustreError: 68325:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1455.694376] LustreError: 68325:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1456.675066] Lustre: 63475:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788904563/real 1788904563] req@00000000b823d677 x1875802264217600/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788904579 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1460.706411] LustreError: 68326:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1460.735980] LustreError: 68326:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1461.727093] Lustre: 63477:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788904568/real 1788904568] req@0000000099364aa2 x1875802264217728/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788904584 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1464.866079] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1465.453430] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1470.990921] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1471.844643] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1481.801877] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1489.133292] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1501.158667] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1501.171277] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1501.189960] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1503.948812] Lustre: server umount lustre-MDT0000 complete [ 1506.789539] LustreError: 68855:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1506.828109] LustreError: 68855:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1507.864088] LustreError: 64546:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788904631 with bad export cookie 18055036625358530594 [ 1507.864852] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1507.878403] LustreError: 64546:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1508.360697] Lustre: server umount lustre-MDT0001 complete [ 1514.946456] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1524.651829] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1527.263179] Lustre: 63478:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788904635/real 1788904635] req@000000002fc0f262 x1875802264245120/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788904651 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1527.297326] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1527.318148] Lustre: Skipped 2 previous similar messages [ 1532.453627] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xfa906026ad29ad85 [ 1532.475254] Lustre: MGC192.168.204.126@tcp: Connection restored to 0@lo (at 0@lo) [ 1532.834794] LustreError: 66403:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1532.928650] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1533.007753] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 1537.258910] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1540.127586] Lustre: 63478:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788904647/real 1788904647] req@000000001e133b8c x1875802264245760/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788904663 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1540.158664] Lustre: 63478:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 1546.215871] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1551.865908] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1552.318253] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1557.051260] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1562.468222] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1568.247204] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1577.455206] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1577.466254] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1577.471032] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1577.471040] Lustre: Skipped 2 previous similar messages [ 1577.522242] Lustre: Skipped 1 previous similar message [ 1582.563173] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1582.588272] Lustre: Skipped 4 previous similar messages [ 1583.224661] Lustre: server umount lustre-MDT0000 complete [ 1587.688586] LustreError: 71011:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1587.698407] LustreError: 71011:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 1587.830269] LustreError: 64545:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788904711 with bad export cookie 18055036625358531973 [ 1587.838199] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1587.850064] LustreError: 64545:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1588.316291] Lustre: server umount lustre-MDT0001 complete [ 1597.268045] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1602.055658] Lustre: DEBUG MARKER: files compared the same [ 1614.862493] Lustre: server umount lustre-OST0000 complete [ 1627.400784] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 1629.300988] Key type lgssc unregistered [ 1629.475104] LNet: 74288:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1629.486928] LNetError: 74288:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1629.509328] LNet: Removed LNI 192.168.204.126@tcp [ 1630.195443] Key type .llcrypt unregistered [ 1630.197415] Key type ._llcrypt unregistered [ 1647.208893] Key type ._llcrypt registered [ 1647.210823] Key type .llcrypt registered [ 1647.270299] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 1647.811526] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1647.878684] alg: No test for adler32 (adler32-zlib) [ 1648.860127] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 1649.010842] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 1650.639239] Key type lgssc registered [ 1651.446797] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1656.520336] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1656.538449] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1657.795175] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1660.141848] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug +malloc all [ 1664.635213] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1664.813671] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1666.918538] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug +malloc all [ 1671.181072] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1674.245884] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1677.989478] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1678.147199] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1681.414879] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug +malloc all [ 1682.292397] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 1685.768819] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1694.689523] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1694.689781] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1694.696437] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1694.707543] Lustre: Skipped 1 previous similar message [ 1694.718744] Lustre: Skipped 1 previous similar message [ 1696.244143] Lustre: server umount lustre-OST0000 complete [ 1698.358626] Lustre: server umount lustre-MDT0000 complete [ 1699.808418] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1699.809217] LustreError: 75684:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1699.814129] Lustre: Skipped 1 previous similar message [ 1699.825411] LustreError: 75684:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1700.168578] LustreError: 75665:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788904824 with bad export cookie 12531713803797799497 [ 1700.170713] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1700.173742] LustreError: 75665:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1706.372740] Lustre: server umount lustre-MDT0001 complete [ 1711.677943] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 1713.026180] Key type lgssc unregistered [ 1713.196690] LNet: 79211:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1713.201553] LNetError: 79211:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1713.210234] LNet: Removed LNI 192.168.204.126@tcp [ 1713.606563] Key type .llcrypt unregistered [ 1713.608554] Key type ._llcrypt unregistered [ 1726.388097] Key type ._llcrypt registered [ 1726.389791] Key type .llcrypt registered [ 1726.448232] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 1727.022792] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1727.071473] alg: No test for adler32 (adler32-zlib) [ 1728.011851] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 1728.166524] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 1729.775187] Key type lgssc registered [ 1730.463859] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1734.591222] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1734.604471] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1750.111468] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1755.233317] LustreError: 80579:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.126@tcp: failed processing log, type 4: rc = -110 [ 1781.855346] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1784.455639] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1787.749292] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1791.419292] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1791.506228] Lustre: *** cfs_fail_loc=706, val=0*** [ 1791.694336] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1791.737760] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 1793.789951] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1796.835973] LustreError: 80605:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1796.844044] LustreError: 80605:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1798.460049] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1798.640968] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1800.689504] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1804.326653] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1806.997296] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1808.868442] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1808.869580] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1808.873661] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1808.877892] Lustre: Skipped 1 previous similar message [ 1813.985172] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1813.988525] Lustre: Skipped 2 previous similar messages [ 1819.106134] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1819.109293] Lustre: Skipped 1 previous similar message [ 1822.175164] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1822.236353] Lustre: server umount lustre-OST0000 complete [ 1824.225181] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1824.225724] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1824.234265] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1824.237124] Lustre: Skipped 1 previous similar message [ 1829.344491] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1829.347121] Lustre: Skipped 1 previous similar message [ 1829.982348] Lustre: server umount lustre-MDT0000 complete [ 1831.417672] LustreError: 80585:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788904955 with bad export cookie 9772469661132339181 [ 1831.421562] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1831.422282] LustreError: 80585:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1831.549409] Lustre: server umount lustre-MDT0001 complete [ 1835.886802] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 1837.042577] Key type lgssc unregistered [ 1837.199441] LNet: 84206:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1837.203405] LNetError: 84206:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1837.217492] LNet: Removed LNI 192.168.204.126@tcp [ 1837.607778] Key type .llcrypt unregistered [ 1837.609704] Key type ._llcrypt unregistered [ 1847.788622] Key type ._llcrypt registered [ 1847.790253] Key type .llcrypt registered [ 1847.830660] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 1848.146386] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1848.159528] alg: No test for adler32 (adler32-zlib) [ 1849.015814] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 1849.107926] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 1850.695186] Key type lgssc registered [ 1851.096119] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1853.892026] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1853.898390] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1856.438287] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1859.112921] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1859.321572] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1860.812901] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1863.981057] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1864.107588] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1866.368544] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1869.216853] LustreError: 86290:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1869.225298] LustreError: 86290:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 1869.572588] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1869.579385] Lustre: MGS: Client 45fc3016-222f-4508-b064-a84af944e439 (at 0@lo) reconnecting [ 1869.582579] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xaf2b187e6c5988d0 [ 1869.586603] Lustre: MGC192.168.204.126@tcp: Connection restored to 0@lo (at 0@lo) [ 1869.695559] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1869.739888] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 1871.288406] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1880.034228] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1880.034688] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1880.039055] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1880.044215] Lustre: Skipped 1 previous similar message [ 1885.154440] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1885.157042] Lustre: Skipped 1 previous similar message [ 1885.718834] Lustre: server umount lustre-OST0000 complete [ 1890.272941] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1890.273304] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1890.273802] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1890.283146] Lustre: Skipped 2 previous similar messages [ 1893.250840] Lustre: server umount lustre-MDT0000 complete [ 1894.641226] Lustre: server umount lustre-MDT0001 complete [ 1895.996545] Lustre: server umount MGS complete [ 1899.763294] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 18:03:42 (1788905022) [ 1909.640527] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [ 1913.356843] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 1926.576032] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1928.249618] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1932.502366] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1932.586923] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 1932.602839] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1932.639179] Lustre: lustre-MDT0001: new disk, initializing [ 1932.664732] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1932.674272] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1932.677250] Lustre: 93445:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 1932.680278] LustreError: 93445:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 1934.082138] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1938.467191] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1938.569904] Lustre: lustre-OST0000: new disk, initializing [ 1938.571774] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1938.574759] Lustre: 94212:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1938.596202] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1940.731815] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1944.661512] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1945.055878] Lustre: 94874:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@0000000053348d6f x1875802819953920/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788905084 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 1947.000510] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1947.006099] Lustre: MGS: Client b2b94083-d802-485d-90ff-9244506a1030 (at 0@lo) reconnecting [ 1947.025331] Lustre: 95259:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1947.033771] LustreError: 93430:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 1947.037886] LustreError: 93430:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 1947.042459] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1947.080720] Lustre: lustre-MDT0000: new disk, initializing [ 1947.141477] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1947.153644] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1948.192298] Lustre: 95286:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@00000000738a65a4 x1875802819959296/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1788905088 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 1948.255648] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1948.685657] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1953.250404] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1953.253490] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1953.320151] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 1973.216695] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1973.219151] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1973.222203] Lustre: Skipped 1 previous similar message [ 1973.224227] Lustre: Skipped 1 previous similar message [ 1978.336839] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1978.338878] Lustre: Skipped 1 previous similar message [ 1978.454321] Lustre: server umount lustre-OST0000 complete [ 1983.456234] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1983.456680] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1983.463433] Lustre: Skipped 1 previous similar message [ 1988.576271] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1988.579301] Lustre: Skipped 3 previous similar messages [ 1994.207108] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1994.251391] Lustre: server umount lustre-MDT0000 complete [ 2001.769468] Lustre: server umount lustre-MDT0001 complete [ 2003.066628] Lustre: server umount MGS complete [ 2006.988398] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 18:05:29 (1788905129) [ 2013.410329] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 2014.471708] Key type lgssc unregistered [ 2014.615366] LNet: 97779:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2014.618413] LNetError: 97779:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2014.629492] LNet: Removed LNI 192.168.204.126@tcp [ 2014.967917] Key type .llcrypt unregistered [ 2014.969563] Key type ._llcrypt unregistered [ 2022.660596] Key type ._llcrypt registered [ 2022.662458] Key type .llcrypt registered [ 2022.707293] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 2023.197295] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2023.249839] alg: No test for adler32 (adler32-zlib) [ 2024.102899] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 2024.195388] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 2025.775246] Key type lgssc registered [ 2026.169482] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2028.896884] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2028.904329] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2030.057412] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2031.460398] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2034.404938] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2034.536900] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2035.991519] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2038.524525] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2040.461479] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2042.991894] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2043.142520] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2045.376092] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2047.944915] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2054.438603] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 2064.867124] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2064.867467] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2064.867867] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2064.872783] Lustre: Skipped 1 previous similar message [ 2066.894740] Lustre: server umount lustre-OST0000 complete [ 2069.984956] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2069.985523] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2069.988817] Lustre: Skipped 1 previous similar message [ 2069.991404] Lustre: Skipped 1 previous similar message [ 2074.338735] Lustre: server umount lustre-MDT0000 complete [ 2075.104663] LustreError: 98978:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2075.110802] LustreError: 98978:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2075.657337] LustreError: 101576:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788905199 with bad export cookie 16272180262135693658 [ 2075.661257] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2075.662056] LustreError: 101576:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2075.779718] Lustre: server umount lustre-MDT0001 complete [ 2079.510282] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 2080.587513] Key type lgssc unregistered [ 2080.718376] LNet: 102528:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2080.721856] LNetError: 102528:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2080.729347] LNet: Removed LNI 192.168.204.126@tcp [ 2081.044761] Key type .llcrypt unregistered [ 2081.046154] Key type ._llcrypt unregistered [ 2087.788163] Key type ._llcrypt registered [ 2087.789546] Key type .llcrypt registered [ 2087.836590] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 2088.182869] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2088.227070] alg: No test for adler32 (adler32-zlib) [ 2089.085637] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 2089.187569] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 2090.775193] Key type lgssc registered [ 2091.224882] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2093.611178] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2093.612364] LustreError: 103589:0:(tgt_mount.c:2659:server_fill_super()) cfs_race id 716 sleeping [ 2093.614922] Lustre: Skipped 1 previous similar message [ 2093.621945] LustreError: 103588:0:(tgt_mount.c:2659:server_fill_super()) cfs_fail_race id 716 waking [ 2093.626080] LustreError: 103589:0:(tgt_mount.c:2659:server_fill_super()) cfs_fail_race id 716 awake: rc=4993 [ 2093.629752] LustreError: 103589:0:(tgt_mount.c:2671:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 2093.634496] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2093.635677] LustreError: 103589:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2094.827877] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2097.680139] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2097.794506] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2099.318884] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2101.224142] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing lsmod [ 2102.339449] LustreError: 104792:0:(tgt_mount.c:2659:server_fill_super()) cfs_race id 716 sleeping [ 2102.340869] LustreError: 104793:0:(tgt_mount.c:2659:server_fill_super()) cfs_fail_race id 716 waking [ 2102.346350] LustreError: 104792:0:(tgt_mount.c:2659:server_fill_super()) cfs_fail_race id 716 awake: rc=4996 [ 2102.349540] LustreError: 104792:0:(tgt_mount.c:2671:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 2102.353737] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2102.354984] LustreError: 104792:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2102.504194] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2103.547202] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2103.548565] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2103.552018] Lustre: Skipped 1 previous similar message [ 2103.556971] Lustre: Skipped 1 previous similar message [ 2107.681502] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2107.682100] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2107.691254] Lustre: Skipped 2 previous similar messages [ 2109.715897] Lustre: server umount lustre-MDT0000 complete [ 2111.121171] LustreError: 103604:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788905234 with bad export cookie 1392038272019263380 [ 2111.124736] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2111.125395] LustreError: 103604:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2111.775574] LustreError: 104056:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000338f4729 x1875803072642688/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2111.836329] Lustre: server umount lustre-MDT0001 complete [ 2123.493272] Lustre: server umount lustre-OST0000 complete [ 2126.472070] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2126.634927] LustreError: 105857:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2126.644992] LustreError: 105857:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2126.660652] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2128.023445] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2131.024959] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2131.163702] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2132.675937] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2135.179544] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2137.137058] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2139.533132] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2139.650029] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2141.553357] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2143.964826] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2151.976399] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 2156.000714] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2156.003479] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2156.008363] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2157.028027] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2162.070063] Lustre: server umount lustre-OST0000 complete [ 2167.264904] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2167.264967] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2167.265508] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2167.265513] Lustre: Skipped 2 previous similar messages [ 2167.278907] Lustre: Skipped 2 previous similar messages [ 2169.627884] Lustre: server umount lustre-MDT0000 complete [ 2171.052672] LustreError: 105837:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788905294 with bad export cookie 1392038272019264066 [ 2171.056550] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2171.058230] LustreError: 105837:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2171.175460] Lustre: server umount lustre-MDT0001 complete [ 2175.235808] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 2176.417365] Key type lgssc unregistered [ 2176.557652] LNet: 109409:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2176.562458] LNetError: 109409:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2176.573536] LNet: Removed LNI 192.168.204.126@tcp [ 2176.918581] Key type .llcrypt unregistered [ 2176.920058] Key type ._llcrypt unregistered [ 2186.030464] Key type ._llcrypt registered [ 2186.031358] Key type .llcrypt registered [ 2186.069105] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 2186.384841] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2186.425589] alg: No test for adler32 (adler32-zlib) [ 2187.282549] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 2187.379276] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 2188.967170] Key type lgssc registered [ 2189.331744] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2191.979822] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2191.986286] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2193.120616] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2194.591308] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2197.483279] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2197.609233] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2198.992731] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2201.443364] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2203.373233] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2205.660238] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2205.770378] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2207.702866] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2210.162877] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2218.086348] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 2222.602423] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 2223.465488] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 2228.708948] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2228.710898] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2228.716493] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2228.718679] LustreError: Skipped 1 previous similar message [ 2230.609803] Lustre: server umount lustre-OST0000 complete [ 2233.314065] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2233.317573] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2233.323850] Lustre: Skipped 1 previous similar message [ 2233.326862] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2233.329689] Lustre: Skipped 1 previous similar message [ 2238.033994] Lustre: server umount lustre-MDT0000 complete [ 2238.944070] LustreError: 110802:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2238.949735] LustreError: 110802:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2239.311687] LustreError: 110782:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788905363 with bad export cookie 4473275751710782089 [ 2239.312291] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2239.316561] LustreError: 110782:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2239.438289] Lustre: server umount lustre-MDT0001 complete [ 2243.227865] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 2244.270695] Key type lgssc unregistered [ 2244.401429] LNet: 114455:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2244.405434] LNetError: 114455:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2244.414558] LNet: Removed LNI 192.168.204.126@tcp [ 2244.725174] Key type .llcrypt unregistered [ 2244.726804] Key type ._llcrypt unregistered [ 2251.661671] Key type ._llcrypt registered [ 2251.663078] Key type .llcrypt registered [ 2251.707327] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 2252.156789] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2252.230115] alg: No test for adler32 (adler32-zlib) [ 2253.093952] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 2253.187759] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 2254.767160] Key type lgssc registered [ 2255.141224] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2257.785052] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2257.792356] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2258.952509] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2260.344956] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2263.154862] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2263.288563] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2264.630270] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2267.144302] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2269.002617] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2271.310357] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2271.422575] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2273.325282] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2275.803451] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2283.753576] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 2289.122716] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2289.124672] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2289.124718] LustreError: Skipped 1 previous similar message [ 2289.130577] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2289.132869] Lustre: Skipped 1 previous similar message [ 2290.825985] Lustre: server umount lustre-OST0000 complete [ 2294.175853] Lustre: server umount lustre-MDT0000 complete [ 2294.239817] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2294.243132] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2294.249039] Lustre: Skipped 1 previous similar message [ 2294.251347] LustreError: 116384:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2294.259364] LustreError: 116384:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2295.494350] LustreError: 115678:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788905419 with bad export cookie 8496775140871184666 [ 2295.498228] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2295.498962] LustreError: 115678:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2295.624557] Lustre: server umount lustre-MDT0001 complete [ 2299.372962] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 2300.393410] Key type lgssc unregistered [ 2300.532595] LNet: 119211:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2300.535096] LNetError: 119211:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2300.542423] LNet: Removed LNI 192.168.204.126@tcp [ 2300.843977] Key type .llcrypt unregistered [ 2300.844951] Key type ._llcrypt unregistered [ 2309.649911] Key type ._llcrypt registered [ 2309.650903] Key type .llcrypt registered [ 2309.687093] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 2309.986759] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2310.019564] alg: No test for adler32 (adler32-zlib) [ 2310.878347] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 2310.958917] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 2312.535165] Key type lgssc registered [ 2312.875730] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2315.358081] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2315.364915] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2316.501521] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2317.814807] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2320.613709] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2320.733376] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2322.145191] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2324.475674] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2326.358979] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2328.563068] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2328.678766] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2330.569724] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2333.032528] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2341.987130] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 2345.820693] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 2347.025483] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 2347.028279] Lustre: Skipped 2 previous similar messages [ 2348.383855] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 2348.386315] Lustre: Skipped 2 previous similar messages [ 2354.169233] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 2354.183858] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 2357.167761] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.204.26@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 2357.171221] Lustre: Skipped 2 previous similar messages [ 2361.143749] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.204.26@tcp 192.168.0.[2,10]@tcp [ 2368.546290] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.204.26/24@tcp in log lustre-MDT0000 [ 2368.549355] Lustre: Skipped 2 previous similar messages [ 2378.238182] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.26/24@tcp [ 2378.239787] Lustre: Skipped 1 previous similar message [ 2382.304538] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2382.308505] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2382.313915] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2383.328813] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2383.330692] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2383.330806] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2386.379026] Lustre: server umount lustre-OST0000 complete [ 2388.448728] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2388.449429] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2388.453271] Lustre: Skipped 1 previous similar message [ 2388.456439] Lustre: Skipped 1 previous similar message [ 2393.568271] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2393.570263] Lustre: Skipped 1 previous similar message [ 2393.750319] Lustre: server umount lustre-MDT0000 complete [ 2395.069319] LustreError: 122442:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788905518 with bad export cookie 2913851888641033340 [ 2395.074172] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2395.189512] Lustre: server umount lustre-MDT0001 complete [ 2398.855596] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 2399.947700] Key type lgssc unregistered [ 2400.071303] LNet: 125939:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2400.073802] LNetError: 125939:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2400.080342] LNet: Removed LNI 192.168.204.126@tcp [ 2400.392173] Key type .llcrypt unregistered [ 2400.393996] Key type ._llcrypt unregistered [ 2410.881411] Key type ._llcrypt registered [ 2410.882570] Key type .llcrypt registered [ 2410.919083] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 2411.291055] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2411.316961] alg: No test for adler32 (adler32-zlib) [ 2412.174514] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 2412.264062] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 2413.847174] Key type lgssc registered [ 2414.231587] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2416.890426] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2416.897398] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2418.055319] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 2418.060480] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2418.066061] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.204.26/24@tcp [ 2419.485577] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2422.262564] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2422.369523] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 2422.377673] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.26/24@tcp [ 2422.393710] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2423.783297] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2426.235331] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2428.100312] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2430.436637] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2430.570786] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2432.627854] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2435.153575] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2442.857040] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 2453.474794] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2453.486171] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2453.491871] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2453.498665] Lustre: Skipped 1 previous similar message [ 2453.509388] Lustre: Skipped 1 previous similar message [ 2458.548852] Lustre: server umount lustre-OST0000 complete [ 2463.714703] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2463.725552] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2463.731041] Lustre: Skipped 1 previous similar message [ 2463.748526] Lustre: Skipped 1 previous similar message [ 2467.937929] Lustre: server umount lustre-MDT0000 complete [ 2468.834615] LustreError: 127474:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2468.860444] LustreError: 127474:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2472.569496] LustreError: 127455:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788905596 with bad export cookie 16853483851023171155 [ 2472.573260] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2472.592370] LustreError: 127455:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2473.290470] Lustre: server umount lustre-MDT0001 complete [ 2487.053896] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 2490.809397] Key type lgssc unregistered [ 2491.227895] LNet: 131081:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2491.240176] LNetError: 131081:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2492.262505] LNet: Removed LNI 192.168.204.126@tcp [ 2493.238654] Key type .llcrypt unregistered [ 2493.243723] Key type ._llcrypt unregistered [ 2516.522667] Key type ._llcrypt registered [ 2516.524630] Key type .llcrypt registered [ 2516.580323] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [ 2525.355596] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 2525.814862] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2525.858148] alg: No test for adler32 (adler32-zlib) [ 2526.775043] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 2526.936409] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 2528.535174] Key type lgssc registered [ 2529.115480] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2547.661468] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2547.671188] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2548.827186] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2548.850592] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2548.902038] Lustre: lustre-MDT0000: new disk, initializing [ 2548.954614] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2548.970922] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2551.609822] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2557.933321] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2558.002439] Lustre: 134459:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2558.054219] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2558.058295] Lustre: Skipped 1 previous similar message [ 2558.113953] Lustre: lustre-MDT0001: new disk, initializing [ 2558.167936] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2558.190303] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2558.197200] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2560.706049] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2566.228604] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2569.264354] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2573.242275] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2573.387410] Lustre: lustre-OST0000: new disk, initializing [ 2573.390629] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2573.396987] Lustre: 136425:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2573.453902] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2576.665949] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2581.897569] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2582.554739] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2582.560515] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2582.606194] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2584.420760] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2584.540135] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2586.931956] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2587.066905] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2592.738453] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2592.739888] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2592.741304] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2592.754599] Lustre: Skipped 1 previous similar message [ 2592.768556] LustreError: Skipped 1 previous similar message [ 2594.344191] Lustre: server umount lustre-OST0000 complete [ 2597.858892] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2597.875109] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2597.879427] Lustre: Skipped 2 previous similar messages [ 2602.543268] Lustre: server umount lustre-MDT0000 complete [ 2604.557568] LustreError: 134450:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788905728 with bad export cookie 9003098180985645100 [ 2604.560076] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2604.562692] LustreError: 134450:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2604.820800] Lustre: server umount lustre-MDT0001 complete [ 2609.661511] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2609.931044] LustreError: 138640:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2609.945060] LustreError: 138640:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2609.977167] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2612.448989] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2615.264860] LustreError: 138641:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2617.487238] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2617.687242] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2620.485508] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2625.040424] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2628.094025] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2632.231141] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2632.464699] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2635.781403] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2640.320899] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2643.017645] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2643.134757] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2645.706726] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2645.843548] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2652.388910] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2652.529531] Lustre: lustre-OST0001: new disk, initializing [ 2652.533432] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2652.539233] Lustre: 142008:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2652.621612] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2656.550864] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2660.401548] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2660.409790] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2660.471906] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2662.259179] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2662.413164] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2665.632236] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2665.793175] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2669.395168] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9a0542eee000.ost_server_uuid 50 [ 2670.357657] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9a0542eee000.ost_server_uuid in FULL state after 0 sec [ 2675.682979] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2675.688573] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2675.691178] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2675.699927] Lustre: Skipped 2 previous similar messages [ 2680.808708] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2680.812882] Lustre: Skipped 2 previous similar messages [ 2681.444734] Lustre: server umount lustre-OST0001 complete [ 2683.872389] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2683.877246] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2685.926121] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2685.926354] LustreError: 140663:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2685.926963] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2685.933539] Lustre: Skipped 2 previous similar messages [ 2685.940796] LustreError: 140663:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2691.051917] LustreError: 140458:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2691.069360] LustreError: 140458:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2696.161826] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2696.163433] LustreError: 140459:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2696.167923] Lustre: Skipped 3 previous similar messages [ 2696.188206] LustreError: 140459:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2698.207159] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2698.298800] Lustre: server umount lustre-OST0000 complete [ 2701.282535] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2701.291176] Lustre: Skipped 2 previous similar messages [ 2706.794778] Lustre: server umount lustre-MDT0000 complete [ 2709.106220] LustreError: 138622:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788905832 with bad export cookie 9003098180985645982 [ 2709.109136] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2709.112933] LustreError: 138622:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2709.325452] Lustre: server umount lustre-MDT0001 complete [ 2720.791813] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 2722.548829] Key type lgssc unregistered [ 2722.746180] LNet: 145008:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2722.758573] LNetError: 145008:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2722.776855] LNet: Removed LNI 192.168.204.126@tcp [ 2723.330910] Key type .llcrypt unregistered [ 2723.334313] Key type ._llcrypt unregistered [ 2748.602733] Key type ._llcrypt registered [ 2748.605607] Key type .llcrypt registered [ 2748.745929] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [ 2773.262530] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 2775.200040] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2775.326417] alg: No test for adler32 (adler32-zlib) [ 2776.770845] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 2777.239806] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 2779.015376] Key type lgssc registered [ 2780.993364] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2838.583292] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2838.649129] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2840.017224] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2840.044370] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2840.101401] Lustre: lustre-MDT0000: new disk, initializing [ 2840.156355] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2840.181326] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2845.184835] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2857.022327] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2857.138565] Lustre: 149166:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2857.246785] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2857.251312] Lustre: Skipped 1 previous similar message [ 2857.349615] Lustre: lustre-MDT0001: new disk, initializing [ 2857.471680] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2857.514439] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2857.519388] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2862.809792] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2873.198377] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2879.279883] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2887.050361] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2887.484642] Lustre: lustre-OST0000: new disk, initializing [ 2887.504668] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2887.521314] Lustre: 151134:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2887.685439] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2892.863913] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2892.885762] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2892.997592] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2894.407594] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2904.313118] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2909.732203] Lustre: Failing over lustre-OST0000 [ 2910.044456] Lustre: server umount lustre-OST0000 complete [ 2928.098099] Lustre: 147385:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788906035/real 1788906035] req@00000000d9515ab3 x1875803794080768/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1788906051 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2928.119104] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2929.184117] Lustre: 147387:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788906037/real 1788906037] req@000000009b6e6a2d x1875803794081536/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788906053 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2929.185765] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2929.226355] Lustre: 147387:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2929.397978] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2929.752964] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2929.782968] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2930.273948] Lustre: 147384:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788906037/real 1788906037] req@000000002f958be4 x1875803794081792/t0(0) o13->lustre-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788906053 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2930.304408] Lustre: 147384:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2934.815148] Lustre: 147384:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788906042/real 1788906042] req@00000000ac5c49ad x1875803794083328/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788906058 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2934.843095] Lustre: 147384:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2936.895875] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2937.915957] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2939.935883] Lustre: 147387:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788906047/real 1788906047] req@000000001574c4fc x1875803794084096/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788906063 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2940.037594] Lustre: lustre-OST0000: Recovery over after 0:03, of 3 clients 3 recovered and 0 were evicted. [ 2940.037866] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2941.298671] Lustre: Failing over lustre-MDT0000 [ 2941.575455] Lustre: server umount lustre-MDT0000 complete [ 2943.013519] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2943.018231] LustreError: 149173:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2943.029499] Lustre: Skipped 1 previous similar message [ 2943.056434] LustreError: 149173:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 2948.064556] LustreError: 149178:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2948.087545] LustreError: 149178:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2953.185572] LustreError: 152862:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2953.219511] LustreError: 152862:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2958.188737] Lustre: 147385:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788906066/real 1788906066] req@00000000533bef68 x1875803794095488/t0(0) o400->MGC192.168.204.126@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788906082 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2958.218937] Lustre: 147385:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 2958.226088] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2958.238453] LustreError: 150205:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2958.253148] LustreError: 150205:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2961.362858] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2963.424597] LustreError: 149178:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2963.443538] LustreError: 149178:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2968.545112] LustreError: 147383:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000009319fae9 x1875803794100224/t0(0) o250->MGC192.168.204.126@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2968.920322] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2968.997776] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2969.631255] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2973.929861] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2974.198022] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2974.208347] Lustre: Skipped 1 previous similar message [ 2974.229813] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 2974.275923] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 2984.423752] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2984.433607] Lustre: Skipped 1 previous similar message [ 2984.443156] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2984.452183] Lustre: Skipped 1 previous similar message [ 2987.711168] Lustre: server umount lustre-OST0000 complete [ 2994.656771] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2994.666921] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2994.682453] Lustre: Skipped 1 previous similar message [ 2994.683540] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2997.722372] Lustre: server umount lustre-MDT0000 complete [ 2999.776295] LustreError: 149178:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2999.808290] LustreError: 149178:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 3001.495858] LustreError: 151942:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788906125 with bad export cookie 6195032698297177198 [ 3001.502954] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3001.513979] LustreError: 151942:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3002.012329] Lustre: server umount lustre-MDT0001 complete [ 3014.491948] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 3017.622226] Key type lgssc unregistered [ 3017.937139] LNet: 154793:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3017.950503] LNetError: 154793:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3017.961976] LNet: Removed LNI 192.168.204.126@tcp [ 3018.948787] Key type .llcrypt unregistered [ 3018.951449] Key type ._llcrypt unregistered [ 3059.666394] Key type ._llcrypt registered [ 3059.670494] Key type .llcrypt registered [ 3059.768977] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 3061.072770] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3061.109623] alg: No test for adler32 (adler32-zlib) [ 3062.197698] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 3062.414930] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 3064.071212] Key type lgssc registered [ 3065.233048] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3073.451106] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3073.475885] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3075.103231] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3079.968482] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3084.273205] LustreError: 156323:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3084.295706] LustreError: 156323:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3089.376679] LustreError: 156322:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3090.647553] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3094.502968] LustreError: 156323:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3095.256892] LustreError: 156351:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 3095.273738] LustreError: 156351:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 3095.283967] LustreError: 156351:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 21, retries 0, failed: rc = -5 [ 3095.785911] Lustre: server umount lustre-MDT0000 complete [ 3111.382799] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [ 3120.912203] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 3157.262479] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3157.458389] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3157.487178] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3157.554708] Lustre: lustre-MDT0000: new disk, initializing [ 3157.621951] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3157.640946] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3162.392336] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3174.874633] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3175.033576] Lustre: 159654:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3175.079604] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3175.082978] Lustre: Skipped 1 previous similar message [ 3175.194552] Lustre: lustre-MDT0001: new disk, initializing [ 3175.289306] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3175.335383] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3175.347274] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3180.051576] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3190.430857] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3196.829592] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3205.034589] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3205.374429] Lustre: lustre-OST0000: new disk, initializing [ 3205.383993] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3205.394315] Lustre: 161622:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3205.500648] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3212.369608] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3214.952530] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3214.960940] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3215.098318] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3222.582601] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3227.608342] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 3229.465179] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 18:25:50 (1788906350) [ 3522.528696] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3522.530123] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3522.555712] Lustre: Skipped 1 previous similar message [ 3522.573596] Lustre: Skipped 2 previous similar messages [ 3527.649751] LustreError: 159665:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3527.681957] LustreError: 159665:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3527.685119] Lustre: server umount lustre-MDT0000 complete [ 3531.333629] LustreError: 159647:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788906655 with bad export cookie 10111710345154462640 [ 3531.334713] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3531.346075] LustreError: 159647:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3532.769114] LustreError: 161046:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3532.777526] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3532.797162] Lustre: Skipped 1 previous similar message [ 3532.801839] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3536.800764] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3537.704524] Lustre: server umount lustre-MDT0001 complete [ 3555.295533] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3555.464309] Lustre: server umount lustre-OST0000 complete [ 3563.743828] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [ 3573.138544] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 3607.752416] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3608.069933] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3608.118706] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3608.240204] Lustre: lustre-MDT0000: new disk, initializing [ 3608.335308] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3608.349711] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3612.818107] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3623.344952] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3623.432836] Lustre: 166292:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3623.495203] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3623.500279] Lustre: Skipped 1 previous similar message [ 3623.591834] Lustre: lustre-MDT0001: new disk, initializing [ 3623.674647] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3623.716902] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3623.732368] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3628.297994] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3637.804489] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3643.337733] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3650.651717] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3651.086587] Lustre: lustre-OST0000: new disk, initializing [ 3651.090899] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3651.097898] Lustre: 168260:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3651.238142] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3658.144361] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3659.901809] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3659.911621] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3660.103322] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3668.159189] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3672.863317] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3673.089300] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3677.764141] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3677.960327] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3680.742281] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3680.742553] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3680.755157] LustreError: Skipped 1 previous similar message [ 3680.772322] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3685.854704] Lustre: server umount lustre-OST0000 complete [ 3690.464480] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3690.474225] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3690.494798] Lustre: Skipped 1 previous similar message [ 3690.509612] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3690.525661] Lustre: Skipped 1 previous similar message [ 3700.193604] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3700.204187] Lustre: Skipped 4 previous similar messages [ 3703.264516] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3703.468139] Lustre: server umount lustre-MDT0000 complete [ 3705.316961] LustreError: 166304:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3705.332959] LustreError: 166304:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3707.372858] LustreError: 166286:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788906831 with bad export cookie 10111710345154621463 [ 3707.381510] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3707.387236] LustreError: 166286:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3707.752306] Lustre: server umount lustre-MDT0001 complete [ 3717.161734] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 18:33:58 (1788906838) [ 3734.921447] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [ 3744.664933] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 3780.478574] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3780.759279] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3780.790261] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3780.889578] Lustre: lustre-MDT0000: new disk, initializing [ 3781.072379] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3781.092164] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3785.063479] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3794.749918] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3794.845435] Lustre: 173016:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3795.136888] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3799.202613] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3808.037299] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3812.930653] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3818.943241] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3819.164172] Lustre: lustre-OST0000: new disk, initializing [ 3819.168855] Lustre: Skipped 1 previous similar message [ 3819.173770] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3819.176904] Lustre: Skipped 2 previous similar messages [ 3819.183198] Lustre: 174997:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3824.043109] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3827.749589] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3827.755235] Lustre: Skipped 1 previous similar message [ 3827.758168] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3827.846676] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3831.240535] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3837.926542] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3837.929993] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3837.948705] Lustre: Skipped 1 previous similar message [ 3837.954857] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3837.964782] Lustre: Skipped 1 previous similar message [ 3842.909572] Lustre: server umount lustre-OST0000 complete [ 3846.624503] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3846.630706] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3846.648692] Lustre: Skipped 1 previous similar message [ 3852.657280] Lustre: server umount lustre-MDT0000 complete [ 3853.281505] LustreError: 174842:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3853.299903] LustreError: 174842:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3855.776954] LustreError: 173817:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788906979 with bad export cookie 10111710345154622352 [ 3855.778353] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3855.784215] LustreError: 173817:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3856.051455] Lustre: server umount lustre-MDT0001 complete [ 3862.842798] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 18:36:24 (1788906984) [ 3876.232853] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [ 3885.181949] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 3911.666145] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3911.816932] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3911.838903] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3911.884220] Lustre: lustre-MDT0000: new disk, initializing [ 3911.927597] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3911.930930] Lustre: Skipped 2 previous similar messages [ 3911.939892] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3914.328418] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3922.245532] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3922.325605] Lustre: 179326:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3922.598375] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3925.859715] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3933.276835] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3937.884960] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3943.463365] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3943.741602] Lustre: 181304:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3949.211121] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3951.246752] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3956.447623] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3961.824843] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3961.833792] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3961.842602] Lustre: Skipped 1 previous similar message [ 3961.847174] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3961.853296] Lustre: Skipped 3 previous similar messages [ 3963.583906] Lustre: server umount lustre-OST0000 complete [ 3972.870890] Lustre: server umount lustre-MDT0000 complete [ 3975.576349] LustreError: 179321:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788907099 with bad export cookie 10111710345154623437 [ 3975.580331] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3975.583562] LustreError: 179321:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3986.015324] Lustre: server umount lustre-MDT0001 complete [ 3993.543265] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 3995.680094] Key type lgssc unregistered [ 3995.910322] LNet: 183158:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3995.917287] LNetError: 183158:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3995.939094] LNet: Removed LNI 192.168.204.126@tcp [ 3996.653170] Key type .llcrypt unregistered [ 3996.655578] Key type ._llcrypt unregistered [ 4014.931917] Key type ._llcrypt registered [ 4014.935319] Key type .llcrypt registered [ 4015.049493] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 4015.701069] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4015.802178] alg: No test for adler32 (adler32-zlib) [ 4016.837232] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 4017.044984] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 4018.735341] Key type lgssc registered [ 4019.667478] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4026.290637] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4026.314112] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4027.616400] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4030.607806] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4037.229686] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4037.661336] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4041.617982] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4048.294514] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4052.926938] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4058.346838] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4058.643621] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4063.113923] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4069.319863] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4074.037041] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4093.408845] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4093.413540] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4093.431668] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4094.435042] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4094.436478] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4094.446199] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4096.776505] Lustre: server umount lustre-OST0000 complete [ 4104.160748] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4104.168280] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4104.179611] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4104.185561] Lustre: Skipped 1 previous similar message [ 4105.885274] Lustre: server umount lustre-MDT0000 complete [ 4109.022894] LustreError: 184537:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788907232 with bad export cookie 798608634766597052 [ 4109.025197] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4109.035605] LustreError: 184537:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4109.286216] Lustre: server umount lustre-MDT0001 complete [ 4117.653322] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 4119.788150] Key type lgssc unregistered [ 4120.021410] LNet: 188110:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4120.025850] LNetError: 188110:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4120.038645] LNet: Removed LNI 192.168.204.126@tcp [ 4120.694141] Key type .llcrypt unregistered [ 4120.699930] Key type ._llcrypt unregistered [ 4138.199931] Key type ._llcrypt registered [ 4138.202049] Key type .llcrypt registered [ 4138.297742] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 4138.894792] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4138.926893] alg: No test for adler32 (adler32-zlib) [ 4139.915723] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 4140.110574] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 4141.751143] Key type lgssc registered [ 4142.633397] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4151.407712] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4151.429201] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4153.107828] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4160.510580] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4162.542224] LustreError: 189504:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4162.571335] LustreError: 189504:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4167.660547] LustreError: 189505:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4171.231336] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4171.524115] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4177.616346] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4187.485839] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4195.509518] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4204.450435] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4204.912279] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4213.404158] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4222.191539] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4222.336892] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 4229.184936] Lustre: lustre-OST0000: Not available for connect from 192.168.204.26@tcp (stopping) [ 4231.649156] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4231.654612] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4231.665659] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4232.673309] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4232.685652] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4232.704098] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4235.034891] Lustre: server umount lustre-OST0000 complete [ 4242.993496] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff9a0543ad3800.ost_server_uuid 50 [ 4245.007646] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 4258.793268] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4258.802723] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4258.824780] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4258.828971] Lustre: Skipped 2 previous similar messages [ 4262.305537] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4264.776789] Lustre: server umount lustre-MDT0000 complete [ 4267.493236] LustreError: 189500:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4267.510950] LustreError: 189500:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4268.581128] LustreError: 189485:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788907392 with bad export cookie 18125170023404592957 [ 4268.587215] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4268.587621] LustreError: 189485:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4268.877427] Lustre: server umount lustre-MDT0001 complete [ 4277.217721] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 18:43:18 (1788907398) [ 4283.668856] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4284.067745] LustreError: 193426:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4284.136572] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4288.365748] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4289.515030] LustreError: 193427:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4295.599555] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4295.912458] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4300.324881] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4309.034607] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4315.489286] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4322.547493] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4322.931367] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4329.083245] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 4329.510653] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4338.562467] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4345.630707] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4345.834626] Lustre: lustre-OST0001: new disk, initializing [ 4345.845631] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4345.861187] Lustre: 196264:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4346.023809] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4351.605591] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4351.618041] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4351.779727] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4352.213879] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4362.545732] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4365.284226] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4365.300477] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4365.317618] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4365.325884] Lustre: Skipped 2 previous similar messages [ 4371.428183] Lustre: server umount lustre-OST0000 complete [ 4372.474014] LustreError: 196584:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4372.497107] LustreError: 196584:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4377.699218] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4377.946759] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 4382.421908] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4382.660776] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 4388.890681] LustreError: 195243:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4388.913576] LustreError: 195243:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 4398.060156] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4398.060329] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4398.072299] LustreError: Skipped 2 previous similar messages [ 4398.092164] Lustre: Skipped 2 previous similar messages [ 4398.101614] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4398.106927] Lustre: Skipped 4 previous similar messages [ 4403.263936] Lustre: server umount lustre-OST0001 complete [ 4413.268912] Lustre: server umount lustre-MDT0000 complete [ 4416.715947] LustreError: 196263:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788907540 with bad export cookie 18125170023404594028 [ 4416.722604] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4416.726618] LustreError: 196263:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 4417.027670] Lustre: server umount lustre-MDT0001 complete [ 4441.010753] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 18:46:02 (1788907562) [ 4447.328918] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4447.535859] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4447.564183] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4447.799181] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4452.600595] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4460.964798] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4461.102721] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4461.156180] Lustre: 199703:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4466.350355] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4474.984763] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4481.236618] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4487.966918] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4488.123309] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4495.004687] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4496.529586] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 4502.831981] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4510.230101] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4510.312293] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 4516.830158] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4525.168106] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4531.183648] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4531.191915] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4531.198300] Lustre: Skipped 3 previous similar messages [ 4531.223221] Lustre: Skipped 3 previous similar messages [ 4534.470071] Lustre: server umount lustre-OST0000 complete [ 4536.302522] LustreError: 203111:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4536.318902] LustreError: 203111:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 12 previous similar messages [ 4551.653415] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4551.661589] LustreError: Skipped 1 previous similar message [ 4554.829185] Lustre: server umount lustre-OST0001 complete [ 4564.539594] Lustre: server umount lustre-MDT0000 complete [ 4567.389782] LustreError: 199698:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788907691 with bad export cookie 18125170023404595435 [ 4567.395182] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4567.399074] LustreError: 199698:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4567.619776] Lustre: server umount lustre-MDT0001 complete [ 4584.685963] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 18:48:26 (1788907706) [ 4595.597178] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [ 4602.100787] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 4626.169202] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4626.342221] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4626.377389] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4626.453317] Lustre: lustre-MDT0000: new disk, initializing [ 4626.507058] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4626.512232] Lustre: Skipped 3 previous similar messages [ 4626.525695] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4629.284209] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4636.478168] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4636.573919] Lustre: 207798:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4636.622791] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4636.629233] Lustre: Skipped 1 previous similar message [ 4636.698670] Lustre: lustre-MDT0001: new disk, initializing [ 4636.800043] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4636.810044] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4639.680049] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4645.545287] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4648.523140] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4652.855886] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4653.062837] Lustre: lustre-OST0000: new disk, initializing [ 4653.065540] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4653.070627] Lustre: 209768:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4653.256422] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4653.261878] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4653.340557] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4656.962757] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4663.140849] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4665.573364] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4665.691521] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4668.179568] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4668.309431] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4674.024211] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4674.025505] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4674.033434] LustreError: Skipped 1 previous similar message [ 4674.035096] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4674.047916] Lustre: Skipped 5 previous similar messages [ 4674.054874] Lustre: Skipped 6 previous similar messages [ 4675.675463] Lustre: server umount lustre-OST0000 complete [ 4684.778841] Lustre: server umount lustre-MDT0000 complete [ 4687.732833] LustreError: 207791:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788907811 with bad export cookie 18125170023404596996 [ 4687.744489] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4687.750249] LustreError: 207791:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4688.024649] Lustre: server umount lustre-MDT0001 complete [ 4694.210430] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4694.546437] LustreError: 211981:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4694.569641] LustreError: 211981:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [ 4697.209984] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4703.133206] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4706.254616] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4711.567219] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4715.592167] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4720.911234] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4725.514517] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4731.039824] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4734.614204] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4740.033817] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 5 sec [ 4743.227502] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4743.459961] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4748.768709] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4748.776738] LustreError: Skipped 1 previous similar message [ 4748.779926] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4748.788847] Lustre: Skipped 2 previous similar messages [ 4748.793787] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4748.798345] Lustre: Skipped 5 previous similar messages [ 4750.995030] Lustre: server umount lustre-OST0000 complete [ 4755.441736] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4758.489744] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4785.478218] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4785.727578] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4785.734045] Lustre: Skipped 5 previous similar messages [ 4789.810974] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4789.819074] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4790.480435] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4796.043508] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4800.048915] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4800.063544] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4804.799708] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 7 sec [ 4813.943530] Lustre: server umount lustre-OST0000 complete [ 4822.577198] Lustre: server umount lustre-MDT0000 complete [ 4825.049512] LustreError: 213822:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788907948 with bad export cookie 18125170023404597878 [ 4825.053780] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4825.059649] LustreError: 213822:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4825.289578] Lustre: server umount lustre-MDT0001 complete [ 4830.857123] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 18:52:33 (1788907953) [ 4835.062957] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4835.327667] LustreError: 217977:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4835.341546] LustreError: 217977:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4837.929838] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4843.041711] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4845.844424] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4850.803547] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4854.156692] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4858.302846] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4862.609961] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4867.710859] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4870.527549] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4870.665937] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4873.716405] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4873.868441] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4878.167168] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4878.275945] Lustre: lustre-OST0001: new disk, initializing [ 4878.278995] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4878.284233] Lustre: 221345:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4882.204255] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4887.080161] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4887.088242] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4887.147047] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4888.332619] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4891.234909] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4891.391940] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4894.135400] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4894.269755] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4897.252793] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4897.253160] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4897.257849] LustreError: Skipped 4 previous similar messages [ 4897.262981] Lustre: Skipped 5 previous similar messages [ 4897.266328] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4897.271273] Lustre: Skipped 7 previous similar messages [ 4901.460988] Lustre: server umount lustre-OST0001 complete [ 4905.504356] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4905.637837] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 4908.716640] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4908.875699] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 4934.979630] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4939.172605] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4940.269534] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 4940.278352] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 4940.278622] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4940.292463] Lustre: Skipped 1 previous similar message [ 4943.816884] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4944.754434] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4960.223143] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4960.284807] Lustre: server umount lustre-OST0001 complete [ 4981.566917] Lustre: server umount lustre-OST0000 complete [ 4990.330878] Lustre: server umount lustre-MDT0000 complete [ 4993.115765] LustreError: 221344:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788908116 with bad export cookie 18125170023404598844 [ 4993.118385] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4993.129982] LustreError: 221344:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4999.532492] Lustre: server umount lustre-MDT0001 complete [ 5016.055401] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 18:55:38 (1788908138) [ 5020.686441] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5020.876808] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5020.905563] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5023.877755] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5029.670530] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5029.759604] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5029.776920] Lustre: 226584:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5032.759376] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5037.174612] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5040.268657] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5044.522477] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5044.668901] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5044.820271] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5044.826207] Lustre: Skipped 7 previous similar messages [ 5048.665546] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5053.705856] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5060.676605] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5060.739313] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5065.039414] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5069.749438] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5070.813601] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5073.871603] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5074.025527] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5077.169586] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5077.346660] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5081.450615] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9a054721e000.ost_server_uuid 50 [ 5082.510889] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9a054721e000.ost_server_uuid in FULL state after 0 sec [ 5083.242884] Lustre: Permanently deactivating lustre-OST0001 [ 5083.259851] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 5085.459505] Lustre: Permanently reactivating lustre-OST0001 [ 5089.959069] Lustre: server umount lustre-OST0001 complete [ 5090.788634] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5090.793717] LustreError: Skipped 2 previous similar messages [ 5095.906840] LustreError: 230124:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5095.921274] LustreError: 230124:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 37 previous similar messages [ 5098.459447] Lustre: server umount lustre-OST0000 complete [ 5106.843745] Lustre: server umount lustre-MDT0000 complete [ 5109.112408] LustreError: 226576:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788908232 with bad export cookie 18125170023404600132 [ 5109.116712] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5109.125408] LustreError: 226576:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5109.366754] Lustre: server umount lustre-MDT0001 complete [ 5126.736137] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 18:57:28 (1788908248) [ 5132.060622] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5132.222839] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5132.249242] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5132.254842] Lustre: Skipped 5 previous similar messages [ 5135.380700] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5141.183952] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5141.246483] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5141.267379] Lustre: 233270:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5144.152623] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5149.780978] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5154.318470] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5159.917922] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5160.079341] Lustre: Permanently deactivating lustre-OST0000 [ 5160.088856] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 5165.747826] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5169.701419] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5171.569530] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5176.453614] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5176.510400] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5176.516400] Lustre: Skipped 1 previous similar message [ 5179.425081] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 5181.722257] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5187.643912] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5189.053566] Lustre: Permanently reactivating lustre-OST0000 [ 5189.071789] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 5189.077519] Lustre: Skipped 2 previous similar messages [ 5198.371631] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5198.379410] Lustre: Skipped 12 previous similar messages [ 5198.385207] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 5198.391614] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5198.402445] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5198.428705] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 5198.448361] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 5203.942165] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5203.946659] Lustre: Skipped 18 previous similar messages [ 5204.188613] Lustre: server umount lustre-OST0001 complete [ 5212.736349] Lustre: server umount lustre-OST0000 complete [ 5221.610293] Lustre: server umount lustre-MDT0000 complete [ 5223.762511] LustreError: 233265:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788908347 with bad export cookie 18125170023404601770 [ 5223.764657] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5223.773857] LustreError: 233265:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5223.986244] Lustre: server umount lustre-MDT0001 complete [ 5230.541924] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 5232.339627] Key type lgssc unregistered [ 5232.548494] LNet: 238149:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5232.560155] LNetError: 238149:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5233.575147] LNet: Removed LNI 192.168.204.126@tcp [ 5234.117811] Key type .llcrypt unregistered [ 5234.121459] Key type ._llcrypt unregistered [ 5249.640289] Key type ._llcrypt registered [ 5249.641890] Key type .llcrypt registered [ 5249.721889] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 5250.247159] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5250.306150] alg: No test for adler32 (adler32-zlib) [ 5251.240300] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 5251.386371] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 5252.991372] Key type lgssc registered [ 5253.698775] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5259.898558] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5259.907704] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5261.220220] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5263.923513] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5269.655421] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5269.735951] Lustre: Found index 1 for lustre-MDT0001, updating log [ 5269.761358] Lustre: 239584:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5269.772597] Lustre: Permanently deactivating lustre-MDT0001 [ 5269.786807] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 5269.988674] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5270.836953] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 5270.840944] LustreError: 239625:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 5270.849396] LustreError: 239625:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 5270.855487] LustreError: 239625:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 9, retries 1, failed: rc = -108 [ 5272.881683] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5277.652294] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5280.829373] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5284.337790] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5284.518496] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5286.566089] LustreError: 241432:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5286.571494] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 5286.575405] LustreError: 241432:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5287.704474] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5291.859665] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5292.007738] LustreError: 241435:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5295.874818] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5296.009528] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5297.072578] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 5297.076672] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 5299.054226] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5302.723364] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5304.764692] Lustre: Permanently reactivating lustre-MDT0001 [ 5304.775399] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 5304.778798] Lustre: Skipped 2 previous similar messages [ 5311.535296] Lustre: lustre-MDT0001: Client e1ef3494-1818-48e8-8dca-6aa3b92534db (at 192.168.204.26@tcp) reconnecting [ 5315.964989] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5317.002634] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5320.574681] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 5320.726789] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 5323.270694] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 5323.417967] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 5325.067598] Lustre: Permanently deactivating lustre-MDT0001 [ 5325.078565] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 5325.085179] Lustre: Skipped 2 previous similar messages [ 5330.961834] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 5335.011418] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5335.013934] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5335.024900] Lustre: Skipped 3 previous similar messages [ 5335.033593] Lustre: Skipped 3 previous similar messages [ 5340.129025] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5340.133853] Lustre: Skipped 3 previous similar messages [ 5340.498445] Lustre: server umount lustre-MDT0000 complete [ 5342.545956] LustreError: 239576:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788908466 with bad export cookie 12774819934340710681 [ 5342.549706] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5342.553583] LustreError: 239576:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5342.719532] Lustre: server umount lustre-MDT0001 complete [ 5350.665755] Lustre: server umount lustre-OST0000 complete [ 5357.268019] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 19:01:19 (1788908479) [ 5364.025846] Lustre: server umount lustre-OST0001 complete [ 5368.355392] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [ 5374.025819] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 5393.647986] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5393.782903] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5393.791343] Lustre: Skipped 2 previous similar messages [ 5393.812235] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5393.863396] Lustre: lustre-MDT0000: new disk, initializing [ 5393.909363] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5393.927457] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5396.467250] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5403.109914] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5403.179878] Lustre: 247851:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5403.229120] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5403.233184] Lustre: Skipped 1 previous similar message [ 5403.286635] Lustre: lustre-MDT0001: new disk, initializing [ 5403.346156] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5403.372068] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5403.378656] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5406.041586] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5412.391506] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5416.323765] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5420.343807] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5420.519519] Lustre: lustre-OST0000: new disk, initializing [ 5420.523067] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5420.527890] Lustre: 249817:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5420.589868] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5424.099436] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5426.209247] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5426.215542] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5426.297111] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5430.494190] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5432.008209] LustreError: 249818:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 5439.500431] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5439.705562] Lustre: lustre-OST0001: new disk, initializing [ 5439.709497] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5439.715792] Lustre: 250931:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5442.091321] LustreError: 249818:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 5442.125837] LustreError: 250538:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 5446.670325] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5449.936260] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5449.953396] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5450.107113] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5452.224355] LustreError: 250538:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 5463.388578] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5466.153527] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5470.520590] Lustre: server umount lustre-OST0001 complete [ 5470.688356] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5470.689852] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5470.724179] LustreError: Skipped 1 previous similar message [ 5470.763150] LustreError: 249802:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5470.783293] LustreError: 249802:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5475.826490] LustreError: 249803:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5475.861361] LustreError: 249803:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5480.935772] LustreError: 250284:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5480.964801] LustreError: 250284:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5491.177124] LustreError: 249802:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5491.197648] LustreError: 249802:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5496.318394] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5496.328875] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5496.330228] Lustre: Skipped 2 previous similar messages [ 5496.339023] Lustre: Skipped 1 previous similar message [ 5499.354961] Lustre: server umount lustre-OST0000 complete [ 5506.017632] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5506.031886] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5506.060538] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5511.648357] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5511.659485] Lustre: Skipped 2 previous similar messages [ 5518.303760] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5518.894521] Lustre: server umount lustre-MDT0000 complete [ 5521.892348] LustreError: 247861:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5521.914635] LustreError: 247861:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5523.548797] LustreError: 247843:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788908647 with bad export cookie 12774819934340713278 [ 5523.555085] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5523.575345] LustreError: 247843:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5530.042627] Lustre: server umount lustre-MDT0001 complete [ 5542.773185] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 5545.502613] Key type lgssc unregistered [ 5545.898821] LNet: 252968:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5545.912385] LNetError: 252968:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5545.951854] LNet: Removed LNI 192.168.204.126@tcp [ 5546.838900] Key type .llcrypt unregistered [ 5546.842107] Key type ._llcrypt unregistered [ 5586.284803] Key type ._llcrypt registered [ 5586.286676] Key type .llcrypt registered [ 5586.389672] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 5587.856169] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5587.896054] alg: No test for adler32 (adler32-zlib) [ 5589.000734] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 5589.285426] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 5591.023242] Key type lgssc registered [ 5592.176705] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5599.836691] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5599.871639] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5601.227419] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5601.248536] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5601.387625] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5605.998783] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5614.770438] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5614.891540] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5614.932475] Lustre: 255176:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5615.210469] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5619.952570] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5628.036773] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5635.115482] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5641.934683] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5642.141872] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5642.320400] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5648.135928] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5651.589299] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 5655.593963] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5707.744474] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5707.762576] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5707.791159] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5708.259344] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5708.264292] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5713.382024] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5713.394181] Lustre: Skipped 3 previous similar messages [ 5713.737570] Lustre: server umount lustre-OST0000 complete [ 5718.333132] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 5732.268272] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5732.692624] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5738.865268] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5738.900332] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5739.268495] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5744.045414] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5744.066046] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5746.789615] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5790.179198] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5790.182302] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5790.193623] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5790.212771] Lustre: Skipped 1 previous similar message [ 5792.865932] Lustre: server umount lustre-OST0000 complete [ 5799.393808] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5799.398404] LustreError: Skipped 1 previous similar message [ 5799.400641] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5799.407741] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5799.411137] Lustre: Skipped 1 previous similar message [ 5802.892436] Lustre: server umount lustre-MDT0000 complete [ 5806.957689] LustreError: 255167:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788908930 with bad export cookie 8206155804354436091 [ 5806.965834] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5806.977870] LustreError: 255167:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5807.613972] Lustre: server umount lustre-MDT0001 complete [ 5820.721227] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 5824.583771] Key type lgssc unregistered [ 5824.984702] LNet: 260598:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5824.990124] LNetError: 260598:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5825.018913] LNet: Removed LNI 192.168.204.126@tcp [ 5825.874314] Key type .llcrypt unregistered [ 5825.876928] Key type ._llcrypt unregistered [ 5852.931399] Key type ._llcrypt registered [ 5852.933933] Key type .llcrypt registered [ 5853.045705] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 5854.250658] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5854.277526] alg: No test for adler32 (adler32-zlib) [ 5855.310732] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 5855.547700] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 5857.191511] Key type lgssc registered [ 5858.276234] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5865.683128] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5865.706988] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5867.005649] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5871.353070] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5876.194357] LustreError: 262016:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5876.209935] LustreError: 262016:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5879.840144] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5880.288099] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5884.910925] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5892.100377] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5898.000049] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5904.483863] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5904.817647] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5911.132854] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5911.983275] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 5918.530769] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5946.853374] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5946.859370] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5946.874721] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5947.876769] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5947.888511] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5947.889893] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5949.415773] Lustre: server umount lustre-OST0000 complete [ 5952.994981] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5952.997257] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5953.005605] Lustre: Skipped 1 previous similar message [ 5953.020148] Lustre: Skipped 2 previous similar messages [ 5958.113023] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5958.119518] Lustre: Skipped 1 previous similar message [ 5963.234857] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5963.240183] Lustre: Skipped 1 previous similar message [ 5967.327366] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5967.485641] Lustre: server umount lustre-MDT0000 complete [ 5968.354736] LustreError: 262016:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5968.380314] LustreError: 262016:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5971.184917] LustreError: 261997:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788909095 with bad export cookie 5444105961524206232 [ 5971.193874] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5971.204661] LustreError: 261997:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5971.734276] Lustre: server umount lustre-MDT0001 complete [ 5982.752518] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 5985.072570] Key type lgssc unregistered [ 5985.343939] LNet: 266322:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5985.348463] LNetError: 266322:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5985.371244] LNet: Removed LNI 192.168.204.126@tcp [ 5986.317472] Key type .llcrypt unregistered [ 5986.319579] Key type ._llcrypt unregistered [ 6003.496702] Key type ._llcrypt registered [ 6003.501188] Key type .llcrypt registered [ 6003.609898] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 6004.284736] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6004.371275] alg: No test for adler32 (adler32-zlib) [ 6005.384919] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 6005.564758] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 6007.215193] Key type lgssc registered [ 6008.243692] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6015.710430] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6015.723143] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6017.063026] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6021.182867] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 6026.210502] LustreError: 267541:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6026.240876] LustreError: 267541:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6028.870497] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6029.294937] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6033.456557] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 6040.436922] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6046.347248] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6052.734067] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6052.973293] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6057.508461] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 6061.162532] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6063.502436] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6071.778612] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6071.785796] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6071.787508] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6071.802148] Lustre: Skipped 1 previous similar message [ 6076.897796] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6076.914052] Lustre: Skipped 2 previous similar messages [ 6077.676642] Lustre: server umount lustre-OST0000 complete [ 6082.018159] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6082.019681] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6082.033258] Lustre: Skipped 1 previous similar message [ 6082.044329] Lustre: Skipped 1 previous similar message [ 6087.137207] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6087.142667] Lustre: Skipped 1 previous similar message [ 6092.256765] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6092.259562] Lustre: Skipped 1 previous similar message [ 6095.327319] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6095.441797] Lustre: server umount lustre-MDT0000 complete [ 6097.376647] LustreError: 269855:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6097.390066] LustreError: 269855:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6098.387606] LustreError: 267523:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788909222 with bad export cookie 13468205757604731979 [ 6098.396700] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6098.406676] LustreError: 267523:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6098.725931] Lustre: server umount lustre-MDT0001 complete [ 6108.099216] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 6110.200311] Key type lgssc unregistered [ 6110.421954] LNet: 271222:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6110.431559] LNetError: 271222:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6110.450947] LNet: Removed LNI 192.168.204.126@tcp [ 6111.086435] Key type .llcrypt unregistered [ 6111.089982] Key type ._llcrypt unregistered [ 6130.631303] Key type ._llcrypt registered [ 6130.633955] Key type .llcrypt registered [ 6130.719932] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 6131.499837] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6131.561088] alg: No test for adler32 (adler32-zlib) [ 6132.530545] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 6132.692231] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 6134.335188] Key type lgssc registered [ 6135.181295] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6141.009296] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6141.020826] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6142.378221] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6145.986774] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 6152.674305] LustreError: 272620:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6152.692798] LustreError: 272620:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6152.738288] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6153.064668] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6156.765704] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 6163.720706] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6169.224211] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6175.587808] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6175.883106] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6181.968586] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 6187.185687] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6189.011606] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6213.098381] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6213.101328] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6213.116263] Lustre: Skipped 1 previous similar message [ 6213.116804] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6213.132587] Lustre: Skipped 1 previous similar message [ 6216.772672] Lustre: server umount lustre-OST0000 complete [ 6223.335398] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6223.359228] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6223.366259] Lustre: Skipped 1 previous similar message [ 6228.452112] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6228.457715] Lustre: Skipped 1 previous similar message [ 6233.569033] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6233.578616] Lustre: Skipped 1 previous similar message [ 6234.080036] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6234.209574] Lustre: server umount lustre-MDT0000 complete [ 6237.060430] LustreError: 272601:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788909360 with bad export cookie 16950656209668796273 [ 6237.067077] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6237.069798] LustreError: 272601:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6237.374224] Lustre: server umount lustre-MDT0001 complete [ 6246.084055] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 6248.385511] Key type lgssc unregistered [ 6248.608831] LNet: 276924:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6248.613895] LNetError: 276924:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6248.626958] LNet: Removed LNI 192.168.204.126@tcp [ 6249.275093] Key type .llcrypt unregistered [ 6249.279453] Key type ._llcrypt unregistered [ 6261.920427] Key type ._llcrypt registered [ 6261.922221] Key type .llcrypt registered [ 6261.990145] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 6262.642864] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6262.690725] alg: No test for adler32 (adler32-zlib) [ 6263.651623] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 6263.832584] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 6265.479216] Key type lgssc registered [ 6266.297939] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6271.487883] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6271.506128] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6272.895824] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6276.058936] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 6282.211575] LustreError: 278138:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6282.226428] LustreError: 278138:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6282.992838] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6283.253078] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6286.442697] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 6291.698841] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6295.429571] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6299.769418] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6300.012658] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6303.899883] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 6309.025632] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6317.424895] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 6327.776541] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6327.794615] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6327.798083] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6327.805676] Lustre: Skipped 1 previous similar message [ 6327.816907] Lustre: Skipped 1 previous similar message [ 6332.899402] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6332.914555] Lustre: Skipped 1 previous similar message [ 6333.485289] Lustre: server umount lustre-OST0000 complete [ 6338.020029] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6338.024508] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6338.039361] Lustre: Skipped 1 previous similar message [ 6342.903750] Lustre: server umount lustre-MDT0000 complete [ 6343.138128] LustreError: 278151:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6343.148087] LustreError: 278151:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6345.707049] LustreError: 280818:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788909469 with bad export cookie 18307068178786464987 [ 6345.708234] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6345.715754] LustreError: 280818:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6345.998499] Lustre: server umount lustre-MDT0001 complete [ 6353.878592] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 6356.155044] Key type lgssc unregistered [ 6356.430832] LNet: 281859:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6356.446444] LNetError: 281859:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6356.477369] LNet: Removed LNI 192.168.204.126@tcp [ 6357.166513] Key type .llcrypt unregistered [ 6357.169061] Key type ._llcrypt unregistered [ 6365.482758] Key type ._llcrypt registered [ 6365.488986] Key type .llcrypt registered [ 6365.570419] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 6380.026202] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [ 6390.637846] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 6391.244791] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6391.312968] alg: No test for adler32 (adler32-zlib) [ 6392.302488] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 6392.492827] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 6394.136709] Key type lgssc registered [ 6394.982751] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6421.324987] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6421.346804] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6422.556249] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6422.582954] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6422.639267] Lustre: lustre-MDT0000: new disk, initializing [ 6422.691766] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6422.703621] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6425.857704] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 6434.435346] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6434.529667] Lustre: 285386:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6434.590764] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6434.597444] Lustre: Skipped 1 previous similar message [ 6434.653496] Lustre: lustre-MDT0001: new disk, initializing [ 6434.767390] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6434.804043] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6434.819343] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6438.296224] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 6445.496691] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6449.263765] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6453.902151] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6454.094840] Lustre: lustre-OST0000: new disk, initializing [ 6454.099747] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6454.106646] Lustre: 287357:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6454.191322] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6458.348190] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 6464.043926] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6464.051786] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6464.126616] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6464.590439] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6467.610407] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6467.786358] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6470.752443] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6470.901460] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6474.208817] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6474.221905] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6474.237442] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6474.726323] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6474.728302] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6474.744190] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6478.478203] Lustre: server umount lustre-OST0000 complete [ 6484.963765] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6484.966302] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6484.974899] Lustre: Skipped 1 previous similar message [ 6487.479292] Lustre: server umount lustre-MDT0000 complete [ 6490.108788] LustreError: 287356:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788909613 with bad export cookie 2235820732204785415 [ 6490.110835] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6490.116522] LustreError: 287356:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6490.358521] Lustre: server umount lustre-MDT0001 complete [ 6496.261268] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 19:20:18 (1788909618) [ 6500.832880] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6501.129719] LustreError: 289766:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6501.142431] LustreError: 289766:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 6501.187696] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6504.083815] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 6506.464224] LustreError: 289767:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6509.571587] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6509.806504] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6512.266123] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 6516.500175] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6519.993588] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6525.036645] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6525.257595] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6529.672166] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 6534.834735] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6544.867622] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6544.868271] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6544.872392] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6544.872401] Lustre: Skipped 1 previous similar message [ 6544.884936] Lustre: Skipped 1 previous similar message [ 6548.139399] Lustre: server umount lustre-OST0000 complete [ 6551.012374] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6551.016663] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6551.026422] LustreError: 289767:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6551.055783] Lustre: server umount lustre-MDT0000 complete [ 6553.844075] LustreError: 289747:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788909677 with bad export cookie 2235820732204786304 [ 6553.847753] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6553.855765] LustreError: 289747:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6554.057871] Lustre: server umount lustre-MDT0001 complete [ 6561.521676] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 6563.565493] Key type lgssc unregistered [ 6563.790084] LNet: 293298:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6563.797178] LNetError: 293298:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6563.820263] LNet: Removed LNI 192.168.204.126@tcp [ 6564.469712] Key type .llcrypt unregistered [ 6564.471481] Key type ._llcrypt unregistered [ 6583.454390] Key type ._llcrypt registered [ 6583.455610] Key type .llcrypt registered [ 6583.544246] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [ 6593.900183] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 6594.544896] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6594.563728] alg: No test for adler32 (adler32-zlib) [ 6595.521692] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 6595.709866] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 6597.351620] Key type lgssc registered [ 6598.208283] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6633.476296] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6633.501892] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6634.667413] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6634.691449] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6634.761863] Lustre: lustre-MDT0000: new disk, initializing [ 6634.822340] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6634.834551] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6638.275673] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 6646.282853] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6646.363459] Lustre: 297098:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6646.400414] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6646.403741] Lustre: Skipped 1 previous similar message [ 6646.486510] Lustre: lustre-MDT0001: new disk, initializing [ 6646.562788] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6646.602258] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6646.609502] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6649.381566] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 6656.017764] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6659.935932] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6664.802499] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6665.029538] Lustre: lustre-OST2710: new disk, initializing [ 6665.034195] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 6665.040066] Lustre: 299067:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6665.099405] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 6668.956577] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 6670.889614] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 6670.904562] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 6670.962836] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 6675.561498] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6680.093524] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6680.183744] Lustre: lustre-OST03e8: new disk, initializing [ 6680.186877] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 6680.193099] Lustre: 300132:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6680.270448] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 6683.845428] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 6688.816629] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6689.332030] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 6689.343035] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 6689.382664] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 6694.763594] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 6694.929379] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6698.087716] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 6698.245692] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6701.267894] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 6701.455112] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6704.677975] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 6704.831464] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6708.193779] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6708.201231] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6708.211258] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6709.728903] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6709.733394] Lustre: lustre-MDT0000-lwp-OST2710: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6709.735913] Lustre: Skipped 2 previous similar messages [ 6709.750136] Lustre: Skipped 2 previous similar messages [ 6713.604915] Lustre: server umount lustre-MDT0000 complete [ 6716.350396] LustreError: 297894:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788909840 with bad export cookie 15191233700681729236 [ 6716.359123] LustreError: 297894:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6716.361948] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6716.597152] Lustre: server umount lustre-MDT0001 complete [ 6729.248827] Lustre: server umount lustre-OST2710 complete [ 6742.169775] Lustre: server umount lustre-OST03e8 complete [ 6753.589800] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [ 6759.793415] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 6781.736416] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6781.866466] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6781.885741] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6781.941736] Lustre: lustre-MDT0000: new disk, initializing [ 6781.999267] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6782.013139] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6784.404761] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 6791.043815] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6791.120429] Lustre: 305378:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6791.161987] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6791.165695] Lustre: Skipped 1 previous similar message [ 6791.216330] Lustre: lustre-MDT0001: new disk, initializing [ 6791.280148] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6791.307909] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6791.317336] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6793.957263] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 6800.344851] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6803.594778] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6807.751323] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6807.928302] Lustre: lustre-OST0000: new disk, initializing [ 6807.931526] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6807.936369] Lustre: 307344:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6807.992845] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6811.851850] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 6814.756516] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6814.767740] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6814.833390] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6818.054913] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6821.034082] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6821.160828] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6823.738152] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6823.859774] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6825.440622] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6825.444454] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6825.460321] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6825.465583] Lustre: Skipped 1 previous similar message [ 6830.051989] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6830.053366] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6830.064112] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6831.215440] Lustre: server umount lustre-OST0000 complete [ 6835.176093] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6835.178294] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6835.189342] Lustre: Skipped 1 previous similar message [ 6835.195759] Lustre: Skipped 1 previous similar message [ 6839.647157] Lustre: server umount lustre-MDT0000 complete [ 6840.289410] LustreError: 306746:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6840.302941] LustreError: 306746:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 6842.168268] LustreError: 308887:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788909966 with bad export cookie 15191233700681730909 [ 6842.170871] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6842.181573] LustreError: 308887:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6842.392276] Lustre: server umount lustre-MDT0001 complete [ 6847.616983] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 19:26:09 (1788909969) [ 6848.531112] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 6849.610752] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 19:26:11 (1788909971) [ 6851.691967] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 6866.706204] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6866.865257] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6866.876328] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6866.987164] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6869.477849] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 6875.283097] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6875.360300] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6875.384626] Lustre: 311199:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6878.236119] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 6896.614365] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6896.725352] Lustre: Denying initial registration attempt from nid 192.168.204.126@tcp, specified as failover [ 6896.731395] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 6896.737621] LustreError: 312651:0:(tgt_mount.c:2703:server_fill_super()) Unable to start targets: -99 [ 6896.741854] LustreError: 312651:0:(tgt_mount.c:2221:server_put_super()) no obd lustre-OST0000 [ 6896.746602] LustreError: 312651:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 6896.791768] Lustre: server umount lustre-OST0000 complete [ 6896.794575] LustreError: 312651:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 6901.216353] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6901.227091] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6901.233865] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6901.239602] Lustre: Skipped 1 previous similar message [ 6906.707047] Lustre: server umount lustre-MDT0000 complete [ 6906.847965] LustreError: 311209:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6906.860334] LustreError: 311209:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6908.843210] LustreError: 311192:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788910032 with bad export cookie 15191233700681731798 [ 6908.846317] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6908.857207] LustreError: 311192:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6909.063134] Lustre: server umount lustre-MDT0001 complete [ 6914.881909] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 6916.437653] Key type lgssc unregistered [ 6916.621051] LNet: 313769:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6916.627537] LNetError: 313769:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6916.643395] LNet: Removed LNI 192.168.204.126@tcp [ 6917.150380] Key type .llcrypt unregistered [ 6917.152408] Key type ._llcrypt unregistered [ 6922.840369] Key type ._llcrypt registered [ 6922.843521] Key type .llcrypt registered [ 6922.902888] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 6923.324812] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6923.334559] alg: No test for adler32 (adler32-zlib) [ 6924.305706] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 6924.501075] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 6926.143538] Key type lgssc registered [ 6927.037936] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6962.978973] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 6968.866326] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6968.878747] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6970.048389] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6970.066676] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6970.207189] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6973.320279] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 6979.224354] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6979.303035] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6979.323517] Lustre: 316507:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6979.503347] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6982.458925] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 6987.459937] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6990.934295] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7008.107690] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7008.240612] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7008.391335] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7012.101749] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 7017.013425] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7023.077133] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7023.080716] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7023.093157] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7023.099182] Lustre: Skipped 1 previous similar message [ 7024.591737] Lustre: server umount lustre-OST0000 complete [ 7028.193258] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7028.199045] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7028.202360] Lustre: Skipped 2 previous similar messages [ 7028.211249] Lustre: Skipped 1 previous similar message [ 7032.904236] Lustre: server umount lustre-MDT0000 complete [ 7035.144320] LustreError: 317250:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788910158 with bad export cookie 11871757917444052019 [ 7035.151564] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7035.154067] LustreError: 317250:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7035.375331] Lustre: server umount lustre-MDT0001 complete [ 7042.704684] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 7044.772599] Key type lgssc unregistered [ 7044.992974] LNet: 320079:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7044.998686] LNetError: 320079:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7046.055204] LNet: Removed LNI 192.168.204.126@tcp [ 7046.776799] Key type .llcrypt unregistered [ 7046.781316] Key type ._llcrypt unregistered [ 7062.349457] Key type ._llcrypt registered [ 7062.353229] Key type .llcrypt registered [ 7062.427515] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 7062.664855] Key type .llcrypt unregistered [ 7062.672608] Key type ._llcrypt unregistered [ 7073.267963] Key type ._llcrypt registered [ 7073.270212] Key type .llcrypt registered [ 7073.354910] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 7074.039473] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7074.051424] alg: No test for adler32 (adler32-zlib) [ 7075.025894] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 7075.197817] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 7076.831206] Key type lgssc registered [ 7077.680925] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7102.019693] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing lnet_if_list [ 7102.794419] LNet: Added LNI 192.168.204.126@tcp1 [8/256/0/180] [ 7110.156742] loop0: detected capacity change from 0 to 204800000 [ 7110.266315] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7110.277329] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7110.290585] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7125.041299] loop0: detected capacity change from 0 to 204800000 [ 7125.153266] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7125.169333] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7125.178515] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7141.029380] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7141.056833] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7142.236654] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7142.251524] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7142.351379] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7145.102115] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 7150.500186] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7150.573160] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7150.591195] Lustre: 323980:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7150.789581] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7153.590516] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 7158.416122] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7162.489970] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7166.464802] loop0: detected capacity change from 0 to 204800000 [ 7183.155130] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7183.315469] Lustre: sv2023-OST0000: new disk, initializing [ 7183.318330] Lustre: srv-sv2023-OST0000: No data found on store. Initialize space. [ 7183.326179] Lustre: 326057:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7183.362649] Lustre: sv2023-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7186.728509] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 7193.050133] loop1: detected capacity change from 0 to 204800000 [ 7209.449813] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7209.531616] Lustre: sv2023-OST0001: new disk, initializing [ 7209.535367] Lustre: srv-sv2023-OST0001: No data found on store. Initialize space. [ 7209.540088] Lustre: 327175:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7209.580883] Lustre: sv2023-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7213.143526] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 7220.962384] Lustre: server umount sv2023-OST0001 complete [ 7226.234287] Lustre: server umount sv2023-OST0000 complete [ 7237.602635] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7237.603747] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7237.610904] Lustre: Skipped 1 previous similar message [ 7237.617026] Lustre: Skipped 1 previous similar message [ 7239.722070] Lustre: server umount lustre-MDT0000 complete [ 7241.744882] LustreError: 323973:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788910365 with bad export cookie 13177507712847028362 [ 7241.748385] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7241.755275] LustreError: 323973:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7241.902655] Lustre: server umount lustre-MDT0001 complete [ 7248.205755] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 7249.686146] Key type lgssc unregistered [ 7249.865736] LNet: 329924:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7249.873646] LNetError: 329924:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7249.878387] LNet: Removed LNI 192.168.204.126@tcp [ 7250.919977] LNet: Removed LNI 192.168.204.126@tcp1 [ 7251.312953] Key type .llcrypt unregistered [ 7251.314687] Key type ._llcrypt unregistered [ 7260.015915] Key type ._llcrypt registered [ 7260.017714] Key type .llcrypt registered [ 7260.069941] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 7260.204968] Key type .llcrypt unregistered [ 7260.207143] Key type ._llcrypt unregistered [ 7269.983147] Key type ._llcrypt registered [ 7269.984421] Key type .llcrypt registered [ 7270.040688] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 7270.565321] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7270.585713] alg: No test for adler32 (adler32-zlib) [ 7271.490704] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 7271.640905] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 7273.271413] Key type lgssc registered [ 7273.836869] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7285.113295] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing lnet_if_list [ 7285.758450] LNet: Added LNI 192.168.204.126@tcp1 [8/256/0/180] [ 7290.730371] loop0: detected capacity change from 0 to 204800000 [ 7290.872379] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7290.879843] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7290.887161] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7305.017458] loop0: detected capacity change from 0 to 204800000 [ 7305.097543] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7305.104910] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7305.116083] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7319.693727] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7319.704792] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7320.822316] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7320.836182] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7320.910867] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7322.951940] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 7327.600829] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7327.658053] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7327.675684] Lustre: 333718:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7327.808342] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7329.718483] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 7332.882956] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7335.216869] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7337.658129] loop0: detected capacity change from 0 to 204800000 [ 7353.254829] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7353.401926] Lustre: sv9991-OST0000: new disk, initializing [ 7353.405188] Lustre: srv-sv9991-OST0000: No data found on store. Initialize space. [ 7353.410926] Lustre: 335796:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7353.446370] Lustre: sv9991-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7356.595763] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 7361.833514] loop1: detected capacity change from 0 to 204800000 [ 7377.580441] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7377.664253] Lustre: sv9991-OST0001: new disk, initializing [ 7377.667255] Lustre: srv-sv9991-OST0001: No data found on store. Initialize space. [ 7377.672216] Lustre: 336912:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7377.703974] Lustre: sv9991-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7380.221267] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 7386.462270] Lustre: server umount sv9991-OST0001 complete [ 7390.723898] Lustre: server umount sv9991-OST0000 complete [ 7399.395556] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7399.398485] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7399.408541] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7400.415792] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7400.418512] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7404.868864] Lustre: server umount lustre-MDT0000 complete [ 7406.848378] LustreError: 333711:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788910530 with bad export cookie 6157158639098301655 [ 7406.853767] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7406.855413] LustreError: 333711:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7407.024299] Lustre: server umount lustre-MDT0001 complete [ 7413.349105] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 7414.910335] Key type lgssc unregistered [ 7415.078570] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.26@tcp [ 7415.096462] LNet: 339662:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7415.103246] LNetError: 339662:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7415.111068] LNet: Removed LNI 192.168.204.126@tcp [ 7416.166085] LNet: Removed LNI 192.168.204.126@tcp1 [ 7416.634567] Key type .llcrypt unregistered [ 7416.636443] Key type ._llcrypt unregistered [ 7426.336719] Key type ._llcrypt registered [ 7426.339202] Key type .llcrypt registered [ 7426.413842] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 7426.569621] Key type .llcrypt unregistered [ 7426.572205] Key type ._llcrypt unregistered [ 7436.255619] Key type ._llcrypt registered [ 7436.259063] Key type .llcrypt registered [ 7436.321810] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 7436.886846] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7436.921108] alg: No test for adler32 (adler32-zlib) [ 7437.863186] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 7438.005792] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 7439.617851] Key type lgssc registered [ 7440.284701] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7452.393269] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing lnet_if_list [ 7453.127656] LNet: Added LNI 192.168.204.126@tcp1 [8/256/0/180] [ 7458.561693] loop0: detected capacity change from 0 to 204800000 [ 7458.740277] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7458.754854] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7458.765118] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7472.940356] loop0: detected capacity change from 0 to 204800000 [ 7473.043215] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7473.058206] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7473.077142] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7488.179166] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7488.190755] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7489.343616] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7489.364938] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7489.478397] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7491.846297] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 7496.441921] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7496.505484] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7496.521923] Lustre: 343478:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7496.691366] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7498.945248] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 7502.906718] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7505.703388] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7508.244466] loop0: detected capacity change from 0 to 204800000 [ 7524.002037] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7524.139847] Lustre: sv5937-OST0000: new disk, initializing [ 7524.143853] Lustre: srv-sv5937-OST0000: No data found on store. Initialize space. [ 7524.147956] Lustre: 345552:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7524.186213] Lustre: sv5937-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7526.980366] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 7534.997156] Lustre: server umount sv5937-OST0000 complete [ 7542.751905] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7542.754908] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7542.762426] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7545.315689] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7545.318483] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7546.483167] Lustre: server umount lustre-MDT0000 complete [ 7548.225782] LustreError: 344219:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788910672 with bad export cookie 4359888281774801222 [ 7548.230361] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7548.235109] LustreError: 344219:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7548.413932] Lustre: server umount lustre-MDT0001 complete [ 7554.663740] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 7556.134311] Key type lgssc unregistered [ 7556.322209] LNet: 347999:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7556.327449] LNetError: 347999:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7556.331717] LNet: Removed LNI 192.168.204.126@tcp [ 7557.349417] LNet: Removed LNI 192.168.204.126@tcp1 [ 7557.782838] Key type .llcrypt unregistered [ 7557.784335] Key type ._llcrypt unregistered [ 7566.798593] Key type ._llcrypt registered [ 7566.800095] Key type .llcrypt registered [ 7566.853712] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 7566.997165] Key type .llcrypt unregistered [ 7566.998563] Key type ._llcrypt unregistered [ 7575.629560] Key type ._llcrypt registered [ 7575.631428] Key type .llcrypt registered [ 7575.718595] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 7576.124580] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7576.133235] alg: No test for adler32 (adler32-zlib) [ 7577.050436] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 7577.177352] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 7578.791254] Key type lgssc registered [ 7579.383718] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7590.908889] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing lnet_if_list [ 7591.523985] LNet: Added LNI 192.168.204.126@tcp1 [8/256/0/180] [ 7597.902743] loop0: detected capacity change from 0 to 204800000 [ 7598.112026] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7598.120641] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7598.138511] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7615.528533] loop0: detected capacity change from 0 to 204800000 [ 7615.728383] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7615.745573] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7615.760109] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7631.947926] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7631.962250] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7633.105868] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7633.130521] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7633.221342] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7635.655552] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 7640.182787] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7640.260239] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7640.294283] Lustre: 351844:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7640.465701] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7642.864437] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 7647.450648] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7650.622084] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7653.280704] loop0: detected capacity change from 0 to 204800000 [ 7669.481702] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7669.667145] Lustre: sv15305-OST0000: new disk, initializing [ 7669.670614] Lustre: srv-sv15305-OST0000: No data found on store. Initialize space. [ 7669.677599] Lustre: 353918:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7669.738369] Lustre: sv15305-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7674.874790] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 7690.366979] Lustre: server umount sv15305-OST0000 complete [ 7698.401261] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7698.403502] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7698.412431] Lustre: Skipped 1 previous similar message [ 7703.521157] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7703.528136] Lustre: Skipped 2 previous similar messages [ 7708.640144] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7708.645837] Lustre: Skipped 1 previous similar message [ 7712.223133] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7712.302729] Lustre: server umount lustre-MDT0000 complete [ 7714.199213] LustreError: 351834:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788910838 with bad export cookie 427175154971541142 [ 7714.203430] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7714.205938] LustreError: 351834:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7714.405660] Lustre: server umount lustre-MDT0001 complete [ 7720.420720] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 7721.745315] Key type lgssc unregistered [ 7721.908291] LNet: 356368:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7721.916951] LNetError: 356368:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7721.922835] LNet: Removed LNI 192.168.204.126@tcp [ 7722.983507] LNet: Removed LNI 192.168.204.126@tcp1 [ 7723.448618] Key type .llcrypt unregistered [ 7723.453327] Key type ._llcrypt unregistered [ 7732.763328] Key type ._llcrypt registered [ 7732.766908] Key type .llcrypt registered [ 7732.864164] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 7733.018289] Key type .llcrypt unregistered [ 7733.021971] Key type ._llcrypt unregistered [ 7742.100780] Key type ._llcrypt registered [ 7742.102586] Key type .llcrypt registered [ 7742.164590] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 7742.652480] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7742.713074] alg: No test for adler32 (adler32-zlib) [ 7743.630679] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 7743.761676] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 7745.383189] Key type lgssc registered [ 7745.953704] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7757.108172] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing lnet_if_list [ 7757.778936] LNet: Added LNI 192.168.204.126@tcp1 [8/256/0/180] [ 7762.963141] loop0: detected capacity change from 0 to 204800000 [ 7763.103401] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7763.113780] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7763.124657] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7777.570180] loop0: detected capacity change from 0 to 204800000 [ 7777.648248] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7777.658597] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7777.668984] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7792.259559] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7792.272986] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7793.415675] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7793.428913] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7793.511146] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7795.634482] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 7800.052390] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7800.120222] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7800.137017] Lustre: 360164:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7800.260200] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7802.349558] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 7806.223672] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7809.109181] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7811.598275] loop0: detected capacity change from 0 to 204800000 [ 7827.554889] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7827.689145] Lustre: sv22-OST0000: new disk, initializing [ 7827.691700] Lustre: srv-sv22-OST0000: No data found on store. Initialize space. [ 7827.695923] Lustre: 362241:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7827.725751] Lustre: sv22-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7830.656346] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 7836.477337] loop1: detected capacity change from 0 to 204800000 [ 7852.333183] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7852.407387] Lustre: sv22-OST0001: new disk, initializing [ 7852.410552] Lustre: srv-sv22-OST0001: No data found on store. Initialize space. [ 7852.414824] Lustre: 363406:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7852.451599] Lustre: sv22-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7855.491717] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 7862.227044] Lustre: server umount sv22-OST0001 complete [ 7866.767136] Lustre: server umount sv22-OST0000 complete [ 7875.040623] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7875.042324] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7875.049184] Lustre: Skipped 1 previous similar message [ 7880.160374] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7880.163856] Lustre: Skipped 2 previous similar messages [ 7885.282138] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7885.285099] Lustre: Skipped 1 previous similar message [ 7888.352111] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7888.429094] Lustre: server umount lustre-MDT0000 complete [ 7890.115953] LustreError: 360158:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788911013 with bad export cookie 13216185124946432513 [ 7890.118972] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7890.121351] LustreError: 360158:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7890.282919] Lustre: server umount lustre-MDT0001 complete [ 7896.127981] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 7897.398729] Key type lgssc unregistered [ 7897.547910] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.26@tcp [ 7897.566669] LNet: 366156:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7897.571073] LNetError: 366156:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7898.591222] LNet: Removed LNI 192.168.204.126@tcp [ 7898.593995] LNetError: 366156:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7898.612273] LNet: Removed LNI 192.168.204.126@tcp1 [ 7899.065927] Key type .llcrypt unregistered [ 7899.071786] Key type ._llcrypt unregistered [ 7907.973138] Key type ._llcrypt registered [ 7907.974249] Key type .llcrypt registered [ 7908.028722] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 7908.170631] Key type .llcrypt unregistered [ 7908.172917] Key type ._llcrypt unregistered [ 7916.892740] Key type ._llcrypt registered [ 7916.894707] Key type .llcrypt registered [ 7916.954417] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 7917.399052] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7917.458588] alg: No test for adler32 (adler32-zlib) [ 7918.403982] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 7918.538572] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 7920.143172] Key type lgssc registered [ 7920.843773] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7930.486267] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing lnet_if_list [ 7931.026763] LNet: Added LNI 192.168.204.126@tcp1 [8/256/0/180] [ 7943.033161] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 7944.259462] Key type lgssc unregistered [ 7944.385865] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.26@tcp [ 7944.407431] LNet: 370087:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7944.411163] LNetError: 370087:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7944.415443] LNet: Removed LNI 192.168.204.126@tcp [ 7944.424702] LNet: Removed LNI 192.168.204.126@tcp1 [ 7944.806592] Key type .llcrypt unregistered [ 7944.808197] Key type ._llcrypt unregistered [ 7952.350084] Key type ._llcrypt registered [ 7952.351879] Key type .llcrypt registered [ 7952.404533] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 7952.531450] Key type .llcrypt unregistered [ 7952.533381] Key type ._llcrypt unregistered [ 7960.405418] Key type ._llcrypt registered [ 7960.407341] Key type .llcrypt registered [ 7960.474159] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 7961.107960] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7961.118295] alg: No test for adler32 (adler32-zlib) [ 7962.021129] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 7962.138223] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 7963.735223] Key type lgssc registered [ 7964.232623] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7972.956127] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing lnet_if_list [ 7973.532041] LNet: Added LNI 192.168.204.126@tcp1 [8/256/0/180] [ 7974.328299] LNet: Added LNI 192.168.204.126@tcp2 [8/256/0/180] [ 7978.879040] loop0: detected capacity change from 0 to 204800000 [ 7978.956851] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7978.963810] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7978.970631] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7992.586801] loop0: detected capacity change from 0 to 204800000 [ 7992.663222] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7992.669482] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7992.676475] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8006.680832] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8006.693575] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8007.773838] LNetError: 371391:0:(peer.c:2149:lnet_peer_add_nid()) Peer 192.168.204.126@tcp NID 192.168.204.126@tcp is already locked with peer 192.168.204.126@tcp [ 8007.830885] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8007.841530] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8007.924202] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8009.715858] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 8013.536033] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8013.592299] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8013.608513] Lustre: 373975:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8013.694053] LNetError: 371391:0:(peer.c:2149:lnet_peer_add_nid()) Peer 192.168.204.126@tcp NID 192.168.204.126@tcp is already locked with peer 192.168.204.126@tcp [ 8013.699495] LNetError: 371391:0:(peer.c:2149:lnet_peer_add_nid()) Skipped 1 previous similar message [ 8013.731826] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8015.423053] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 8016.112123] LNetError: 371391:0:(peer.c:2149:lnet_peer_add_nid()) Peer 192.168.204.126@tcp NID 192.168.204.126@tcp is already locked with peer 192.168.204.126@tcp [ 8016.117284] LNetError: 371391:0:(peer.c:2149:lnet_peer_add_nid()) Skipped 2 previous similar messages [ 8018.415413] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8020.650020] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8022.802799] loop0: detected capacity change from 0 to 204800000 [ 8038.307764] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8038.416610] Lustre: sv12813-OST0000: new disk, initializing [ 8038.419353] Lustre: srv-sv12813-OST0000: No data found on store. Initialize space. [ 8038.423534] Lustre: 376049:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8038.450438] Lustre: sv12813-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8040.958423] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 8045.926937] loop1: detected capacity change from 0 to 204800000 [ 8061.158524] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8061.235770] Lustre: sv12813-OST0001: new disk, initializing [ 8061.238907] Lustre: srv-sv12813-OST0001: No data found on store. Initialize space. [ 8061.243750] Lustre: 377217:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8061.272482] Lustre: sv12813-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8063.827844] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 8070.126580] Lustre: server umount sv12813-OST0001 complete [ 8074.405506] Lustre: server umount sv12813-OST0000 complete [ 8080.351909] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8080.354860] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8080.360688] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8082.913150] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8082.913626] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8085.660067] Lustre: server umount lustre-MDT0000 complete [ 8087.145463] LustreError: 373965:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788911210 with bad export cookie 1654706485780148383 [ 8087.150447] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8087.151166] LustreError: 373965:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8087.279736] Lustre: server umount lustre-MDT0001 complete [ 8092.437313] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 8093.678220] Key type lgssc unregistered [ 8093.832595] LNet: 379966:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8093.836267] LNetError: 379966:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8093.839576] LNet: Removed LNI 192.168.204.126@tcp [ 8094.879148] LNet: Removed LNI 192.168.204.126@tcp1 [ 8094.882355] LNetError: 379966:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8094.887627] LNetError: 379966:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 8094.898796] LNet: Removed LNI 192.168.204.126@tcp2 [ 8095.333876] Key type .llcrypt unregistered [ 8095.336106] Key type ._llcrypt unregistered [ 8103.369217] Key type ._llcrypt registered [ 8103.370759] Key type .llcrypt registered [ 8103.419849] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 8103.550473] Key type .llcrypt unregistered [ 8103.554449] Key type ._llcrypt unregistered [ 8111.161877] Key type ._llcrypt registered [ 8111.163039] Key type .llcrypt registered [ 8111.205170] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 8111.679944] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8111.702708] alg: No test for adler32 (adler32-zlib) [ 8112.585159] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 8112.683575] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 8114.279153] Key type lgssc registered [ 8114.671773] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8122.736937] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing lnet_if_list [ 8123.274848] LNet: Added LNI 192.168.204.126@tcp1 [8/256/0/180] [ 8126.894686] loop0: detected capacity change from 0 to 204800000 [ 8126.993172] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8126.999364] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8127.005527] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8140.448417] loop0: detected capacity change from 0 to 204800000 [ 8140.583310] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8140.590716] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8140.597447] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8154.261168] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8154.268179] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8155.356817] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8155.366084] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8155.433567] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8156.932401] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 8159.961209] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8159.999764] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8160.012161] Lustre: 383809:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8160.131565] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8161.589917] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 8164.698958] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8166.877448] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8168.860277] loop0: detected capacity change from 0 to 204800000 [ 8183.906836] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8184.001070] Lustre: sv23770-OST0000: new disk, initializing [ 8184.003507] Lustre: srv-sv23770-OST0000: No data found on store. Initialize space. [ 8184.006813] Lustre: 385882:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8184.031710] Lustre: sv23770-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8186.200768] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 8190.630335] loop1: detected capacity change from 0 to 204800000 [ 8205.921135] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8205.972776] Lustre: sv23770-OST0001: new disk, initializing [ 8205.975204] Lustre: srv-sv23770-OST0001: No data found on store. Initialize space. [ 8205.978080] Lustre: 386998:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8205.999495] Lustre: sv23770-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8208.345327] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 8213.931764] Lustre: server umount sv23770-OST0001 complete [ 8217.670466] Lustre: server umount sv23770-OST0000 complete [ 8226.784800] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8226.784813] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8226.785571] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8226.794077] Lustre: Skipped 1 previous similar message [ 8228.503900] Lustre: server umount lustre-MDT0000 complete [ 8229.838366] LustreError: 383800:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788911353 with bad export cookie 3908009415783302185 [ 8229.841447] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8229.843453] LustreError: 383800:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8229.958518] Lustre: server umount lustre-MDT0001 complete [ 8235.202872] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 8236.335950] Key type lgssc unregistered [ 8236.442660] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.26@tcp [ 8236.461386] LNet: 389748:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8236.465204] LNetError: 389748:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8236.468434] LNet: Removed LNI 192.168.204.126@tcp [ 8237.477787] LNet: Removed LNI 192.168.204.126@tcp1 [ 8237.830267] Key type .llcrypt unregistered [ 8237.831617] Key type ._llcrypt unregistered [ 8244.288196] Key type ._llcrypt registered [ 8244.289729] Key type .llcrypt registered [ 8244.337303] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 8244.448276] Key type .llcrypt unregistered [ 8244.449666] Key type ._llcrypt unregistered [ 8246.446865] Key type ._llcrypt registered [ 8246.448329] Key type .llcrypt registered [ 8246.485446] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 8246.589272] Key type .llcrypt unregistered [ 8246.590441] Key type ._llcrypt unregistered [ 8251.086935] Key type ._llcrypt registered [ 8251.088558] Key type .llcrypt registered [ 8251.136295] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 8251.241365] Key type .llcrypt unregistered [ 8251.243058] Key type ._llcrypt unregistered [ 8257.662138] Key type ._llcrypt registered [ 8257.663348] Key type .llcrypt registered [ 8257.700385] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 8257.814298] Key type .llcrypt unregistered [ 8257.815848] Key type ._llcrypt unregistered [ 8264.795829] Key type ._llcrypt registered [ 8264.797272] Key type .llcrypt registered [ 8264.838348] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 8265.338107] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8265.369877] alg: No test for adler32 (adler32-zlib) [ 8266.266247] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 8266.393943] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 8267.991192] Key type lgssc registered [ 8268.492176] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8277.958404] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing lnet_if_list [ 8278.504442] LNet: Added LNI 192.168.204.126@tcp1 [8/256/0/180] [ 8282.468947] loop0: detected capacity change from 0 to 204800000 [ 8282.530780] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8282.535716] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8282.540883] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8296.073711] loop0: detected capacity change from 0 to 204800000 [ 8296.162556] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8296.168775] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8296.174892] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8309.664624] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8309.670247] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8310.756802] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8310.765493] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8310.832976] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8312.296690] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 8315.442969] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8315.473737] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8315.484158] Lustre: 395182:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8315.591575] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8317.107973] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 8319.557255] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8321.422643] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8323.169329] loop0: detected capacity change from 0 to 204800000 [ 8338.145397] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8338.240160] Lustre: sv19075-OST0000: new disk, initializing [ 8338.242602] Lustre: srv-sv19075-OST0000: No data found on store. Initialize space. [ 8338.246543] Lustre: 397256:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8338.274258] Lustre: sv19075-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8340.383554] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 8343.721707] Lustre: server umount sv19075-OST0000 complete [ 8348.318370] loop0: detected capacity change from 0 to 204800000 [ 8352.418051] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8352.523290] Lustre: *** cfs_fail_loc=727, val=1*** [ 8352.528636] Lustre: sv19075-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8354.658504] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 8360.318400] Lustre: server umount sv19075-OST0000 complete [ 8367.071952] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8367.074942] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8367.080456] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8367.585036] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8367.592055] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8372.704478] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8372.707017] Lustre: Skipped 1 previous similar message [ 8377.824476] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8377.826847] Lustre: Skipped 1 previous similar message [ 8379.359120] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8379.423778] Lustre: server umount lustre-MDT0000 complete [ 8380.761289] LustreError: 400669:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788911504 with bad export cookie 12932756801832802466 [ 8380.765744] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8380.766383] LustreError: 400669:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8380.890164] Lustre: server umount lustre-MDT0001 complete [ 8384.885871] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 8385.954847] Key type lgssc unregistered [ 8386.095414] LNet: 401420:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8386.098854] LNetError: 401420:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8386.102489] LNet: Removed LNI 192.168.204.126@tcp [ 8387.173460] LNet: Removed LNI 192.168.204.126@tcp1 [ 8387.516427] Key type .llcrypt unregistered [ 8387.518392] Key type ._llcrypt unregistered [ 8396.037080] Key type ._llcrypt registered [ 8396.038730] Key type .llcrypt registered [ 8396.079802] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 8396.529922] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8396.541933] alg: No test for adler32 (adler32-zlib) [ 8397.401040] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 8397.491812] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 8399.079220] Key type lgssc registered [ 8399.491996] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8402.534547] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8402.541713] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8403.695527] LNetError: 402026:0:(lib-move.c:2123:lnet_handle_find_routed_path()) peer 192.168.204.126@tcp1 has no available nets [ 8403.711426] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8405.400839] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 8409.122637] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8409.218354] LNetError: 402026:0:(lib-move.c:2123:lnet_handle_find_routed_path()) peer 192.168.204.126@tcp1 has no available nets [ 8409.223176] LNetError: 402026:0:(lib-move.c:2123:lnet_handle_find_routed_path()) Skipped 2 previous similar messages [ 8409.261777] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8410.816239] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 8413.476726] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8415.689743] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8420.519214] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8420.616595] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8420.679516] LNetError: 402026:0:(lib-move.c:2123:lnet_handle_find_routed_path()) peer 192.168.204.126@tcp1 has no available nets [ 8420.683285] LNetError: 402026:0:(lib-move.c:2123:lnet_handle_find_routed_path()) Skipped 2 previous similar messages [ 8420.697476] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8423.005668] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 8425.844032] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8428.011262] LNetError: 402026:0:(lib-move.c:2123:lnet_handle_find_routed_path()) peer 192.168.204.126@tcp1 has no available nets [ 8428.015483] LNetError: 402026:0:(lib-move.c:2123:lnet_handle_find_routed_path()) Skipped 2 previous similar messages [ 8435.168138] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8435.171690] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8435.177635] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8437.283913] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.26@tcp (stopping) [ 8438.242553] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8438.249039] Lustre: Skipped 1 previous similar message [ 8440.605384] Lustre: server umount lustre-MDT0000 complete [ 8442.088383] LustreError: 404548:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788911565 with bad export cookie 10792315020516562317 [ 8442.092340] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8442.094402] LustreError: 404548:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8442.393988] LustreError: 402712:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8442.403798] LustreError: 402712:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8442.406761] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.26@tcp (stopping) [ 8442.417154] Lustre: Skipped 3 previous similar messages [ 8443.359852] LustreError: 404558:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8443.360420] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8447.392214] LustreError: 402712:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8447.392614] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8447.405207] Lustre: Skipped 1 previous similar message [ 8448.307660] Lustre: server umount lustre-MDT0001 complete [ 8450.626137] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 8453.582189] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8466.911772] LustreError: 402027:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000f998b659 x1875809687071744/t0(0) o250->MGC192.168.204.126@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8467.026206] LNetError: 402026:0:(lib-move.c:2123:lnet_handle_find_routed_path()) peer 192.168.204.126@tcp1 has no available nets [ 8467.031263] LNetError: 402026:0:(lib-move.c:2123:lnet_handle_find_routed_path()) Skipped 5 previous similar messages [ 8467.040276] LustreError: 404541:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8467.047456] LustreError: 404541:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8467.068072] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8467.095059] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 8468.572147] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 8471.879697] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8472.030495] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8473.062350] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 8473.427067] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 8475.923543] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8477.159683] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 8478.518432] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 8480.443950] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8481.004313] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8482.783869] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8482.787768] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8482.792241] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8482.794213] Lustre: Skipped 1 previous similar message [ 8487.394570] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8487.399565] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8488.465969] Lustre: server umount lustre-OST0000 complete [ 8492.516175] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8492.517558] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8492.522883] Lustre: Skipped 3 previous similar messages [ 8504.287130] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8504.347141] Lustre: server umount lustre-MDT0000 complete [ 8505.739187] LustreError: 404548:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788911629 with bad export cookie 10792315020516569562 [ 8505.741100] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8505.744881] LustreError: 404548:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8505.866642] Lustre: server umount lustre-MDT0001 complete [ 8509.531301] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 8510.683440] Key type lgssc unregistered [ 8510.816445] LNet: 408988:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8510.818845] LNetError: 408988:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8510.830395] LNet: Removed LNI 192.168.204.126@tcp [ 8511.150698] Key type .llcrypt unregistered [ 8511.151787] Key type ._llcrypt unregistered [ 8521.011807] Key type ._llcrypt registered [ 8521.013039] Key type .llcrypt registered [ 8521.052507] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 8521.496685] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8521.506069] alg: No test for adler32 (adler32-zlib) [ 8522.354536] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 8522.437601] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 8524.015190] Key type lgssc registered [ 8524.365795] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8526.939993] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8526.947323] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8528.098859] LNetError: 409691:0:(lib-move.c:2123:lnet_handle_find_routed_path()) peer 192.168.204.126@tcp1 has no available nets [ 8528.119257] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8529.528328] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 8532.536989] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8532.628531] LNetError: 409691:0:(lib-move.c:2123:lnet_handle_find_routed_path()) peer 192.168.204.126@tcp1 has no available nets [ 8532.631100] LNetError: 409691:0:(lib-move.c:2123:lnet_handle_find_routed_path()) Skipped 4 previous similar messages [ 8532.668216] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8534.037551] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 8536.480631] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8538.313052] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8542.369497] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8542.378453] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8545.432885] Lustre: server umount lustre-MDT0000 complete [ 8546.745160] LustreError: 410361:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788911670 with bad export cookie 2029859315683666697 [ 8546.747669] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8546.749170] LustreError: 410361:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8546.862865] Lustre: server umount lustre-MDT0001 complete [ 8551.865271] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8551.929887] LNetError: 409691:0:(lib-move.c:2123:lnet_handle_find_routed_path()) peer 192.168.204.126@tcp1 has no available nets [ 8551.934122] LNetError: 409691:0:(lib-move.c:2123:lnet_handle_find_routed_path()) Skipped 5 previous similar messages [ 8552.001657] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8552.017705] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8552.095366] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8553.499925] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 8556.480732] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8556.516792] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8556.528861] Lustre: 412965:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8556.592994] LNetError: 409691:0:(lib-move.c:2123:lnet_handle_find_routed_path()) peer 192.168.204.126@tcp1 has no available nets [ 8556.596220] LNetError: 409691:0:(lib-move.c:2123:lnet_handle_find_routed_path()) Skipped 1 previous similar message [ 8556.630042] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8558.150757] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 8560.669246] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8562.597207] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8567.394575] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8567.468837] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 8567.473439] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 8567.478250] LustreError: 414847:0:(tgt_mount.c:2703:server_fill_super()) Unable to start targets: -2 [ 8567.480923] LustreError: 414847:0:(tgt_mount.c:2221:server_put_super()) no obd lustre-OST0000 [ 8567.483309] LustreError: 414847:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 8567.513369] Lustre: server umount lustre-OST0000 complete [ 8567.515256] LustreError: 414847:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 8572.578280] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8572.647320] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8572.708468] LNetError: 409691:0:(lib-move.c:2123:lnet_handle_find_routed_path()) peer 192.168.204.126@tcp1 has no available nets [ 8572.712581] LNetError: 409691:0:(lib-move.c:2123:lnet_handle_find_routed_path()) Skipped 2 previous similar messages [ 8572.729761] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8574.820327] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 8577.328234] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8580.045502] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8580.075977] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 8580.099109] Lustre: lustre-OST0001: new disk, initializing [ 8580.100667] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8580.102842] Lustre: 416377:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8581.366991] LNetError: 409691:0:(lib-move.c:2123:lnet_handle_find_routed_path()) peer 192.168.204.126@tcp1 has no available nets [ 8581.369691] LNetError: 409691:0:(lib-move.c:2123:lnet_handle_find_routed_path()) Skipped 3 previous similar messages [ 8581.395080] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8581.399321] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8581.427804] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 8581.456473] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8582.066909] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 8584.606949] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8586.722374] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8586.723297] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8586.725856] LustreError: Skipped 1 previous similar message [ 8586.732399] Lustre: Skipped 1 previous similar message [ 8586.734768] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8586.737882] Lustre: Skipped 2 previous similar messages [ 8587.712579] Lustre: server umount lustre-OST0001 complete [ 8591.841324] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8591.842940] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8591.843174] LustreError: 415312:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8591.843186] LustreError: 415312:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8591.844425] LustreError: Skipped 1 previous similar message [ 8591.850427] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8591.861504] Lustre: Skipped 1 previous similar message [ 8595.151081] Lustre: server umount lustre-OST0000 complete [ 8596.960496] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8596.961347] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8596.965087] Lustre: Skipped 2 previous similar messages [ 8602.080850] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8602.084120] Lustre: Skipped 2 previous similar messages [ 8602.653553] Lustre: server umount lustre-MDT0000 complete [ 8604.039395] LustreError: 412957:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788911727 with bad export cookie 2029859315683667236 [ 8604.042714] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8604.044926] LustreError: 412957:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8604.162104] Lustre: server umount lustre-MDT0001 complete [ 8607.987685] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 8609.098141] Key type lgssc unregistered [ 8609.227431] LNet: 418337:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8609.230111] LNetError: 418337:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8609.239334] LNet: Removed LNI 192.168.204.126@tcp [ 8609.580897] Key type .llcrypt unregistered [ 8609.582271] Key type ._llcrypt unregistered [ 8640.620868] Key type ._llcrypt registered [ 8640.622062] Key type .llcrypt registered [ 8640.660466] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [ 8647.240461] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 8647.728949] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8647.738930] alg: No test for adler32 (adler32-zlib) [ 8648.617808] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 8648.731321] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 8650.313162] Key type lgssc registered [ 8650.728367] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8663.591684] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8663.602420] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8664.698565] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8664.711664] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8664.749758] Lustre: lustre-MDT0000: new disk, initializing [ 8664.772580] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8664.784593] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8666.225926] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 8670.478165] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8670.524050] Lustre: 423242:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8670.549646] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8670.552108] Lustre: Skipped 1 previous similar message [ 8670.591103] Lustre: lustre-MDT0001: new disk, initializing [ 8670.625194] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8670.638561] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8670.642794] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8672.206741] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 8676.406089] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8678.474796] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8680.998491] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8681.112447] Lustre: lustre-OST0000: new disk, initializing [ 8681.114368] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8681.118534] Lustre: 425209:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8681.156394] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8681.566781] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8681.571117] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8681.621106] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8683.309944] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 8687.347840] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8688.987989] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8689.078198] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8690.656673] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8690.744098] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8692.194436] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8692.195039] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8692.198539] LustreError: Skipped 1 previous similar message [ 8692.204761] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8692.208551] Lustre: Skipped 1 previous similar message [ 8697.315856] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8697.318760] Lustre: Skipped 1 previous similar message [ 8697.562149] Lustre: server umount lustre-OST0000 complete [ 8701.408230] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8701.411568] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8701.417939] Lustre: Skipped 1 previous similar message [ 8701.420286] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8702.433232] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8705.128348] Lustre: server umount lustre-MDT0000 complete [ 8706.582776] LustreError: 423235:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788911830 with bad export cookie 15133190575121463143 [ 8706.587104] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8706.590619] LustreError: 423235:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8706.713736] Lustre: server umount lustre-MDT0001 complete [ 8709.946659] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 19:57:12 (1788911832) [ 8712.773971] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 19:57:15 (1788911835) [ 8716.148129] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8716.366982] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8718.060055] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 8721.377052] LustreError: 427961:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8721.383081] LustreError: 427961:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8721.693569] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8721.884247] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8723.527505] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 8726.251474] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8728.251248] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8730.714417] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8730.856544] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8733.006862] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 8735.483476] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8749.335816] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 8749.891551] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 8750.484964] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 8751.075491] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 8751.705979] Lustre: Failing over lustre-MDT0000 [ 8751.888447] Lustre: server umount lustre-MDT0000 complete [ 8752.609711] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8752.610239] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8752.611070] LustreError: 429304:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8752.613770] Lustre: Skipped 2 previous similar messages [ 8756.252732] LustreError: 427956:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8756.258931] LustreError: 427956:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8761.364795] LustreError: 429304:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8761.369891] LustreError: 429304:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 8764.918129] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8764.963251] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8765.052507] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8765.075243] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 8766.438279] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 8766.485822] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 8770.534948] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 8770.547040] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 8770.564703] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 8772.616091] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8773.229160] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8775.169371] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 8775.885962] Lustre: Failing over lustre-MDT0000 [ 8776.056884] Lustre: server umount lustre-MDT0000 complete [ 8776.733119] LustreError: 427957:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8776.739111] LustreError: 427957:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 8780.769836] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8780.770379] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8780.774335] Lustre: Skipped 1 previous similar message [ 8785.889020] LustreError: 427962:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8785.894735] LustreError: 427962:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 8789.122948] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8789.172920] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8789.293210] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 8790.659878] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 8792.084445] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 8794.599489] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8794.601963] Lustre: Skipped 2 previous similar messages [ 8794.611743] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 8794.628554] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 8796.606325] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8797.213933] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8799.715469] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8799.716504] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8799.717411] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8799.717422] Lustre: Skipped 2 previous similar messages [ 8799.720959] Lustre: Skipped 3 previous similar messages [ 8804.833203] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8804.836044] Lustre: Skipped 4 previous similar messages [ 8805.778707] Lustre: server umount lustre-MDT0000 complete [ 8807.082077] LustreError: 429802:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788911930 with bad export cookie 15133190575121465824 [ 8807.084984] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8807.085367] LustreError: 429802:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8807.201928] Lustre: server umount lustre-MDT0001 complete [ 8818.791733] Lustre: server umount lustre-OST0000 complete [ 8824.263230] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8824.431181] LustreError: 434311:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8824.438631] LustreError: 434311:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8824.459118] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8824.462066] Lustre: Skipped 1 previous similar message [ 8825.803695] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 8828.975367] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8830.659773] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 8833.333797] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8835.252547] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8837.604527] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8839.921426] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 8842.600736] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8850.022578] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 8854.496069] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8854.499516] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8854.505277] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8856.404757] Lustre: server umount lustre-OST0000 complete [ 8859.862308] Lustre: server umount lustre-MDT0000 complete [ 8860.128265] LustreError: 434312:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8861.131084] LustreError: 434291:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788911984 with bad export cookie 15133190575121466545 [ 8861.133985] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8861.135564] LustreError: 434291:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8861.243954] Lustre: server umount lustre-MDT0001 complete [ 8864.868395] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 8865.975531] Key type lgssc unregistered [ 8866.122470] LNet: 437858:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8866.125662] LNetError: 437858:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8866.132412] LNet: Removed LNI 192.168.204.126@tcp [ 8866.446323] Key type .llcrypt unregistered [ 8866.447797] Key type ._llcrypt unregistered [ 8876.386765] Key type ._llcrypt registered [ 8876.388215] Key type .llcrypt registered [ 8876.425396] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 8876.862192] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8876.872055] alg: No test for adler32 (adler32-zlib) [ 8877.735622] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 8877.844149] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 8879.431101] Key type lgssc registered [ 8879.831925] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8882.550376] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8882.557611] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8883.707939] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8885.081948] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 8888.100988] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8888.251082] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8889.791283] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 8892.438385] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8894.352474] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8896.679610] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8896.794853] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8898.754398] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 8901.207271] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8909.099897] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 8919.007853] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8919.010688] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8919.015921] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8919.521445] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8919.522842] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8919.526634] Lustre: Skipped 1 previous similar message [ 8921.999985] Lustre: server umount lustre-MDT0000 complete [ 8923.271248] LustreError: 441948:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788912047 with bad export cookie 1857386101790419303 [ 8923.273546] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8923.275357] LustreError: 441948:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8923.395201] Lustre: server umount lustre-MDT0001 complete [ 8935.005863] Lustre: server umount lustre-OST0000 complete [ 8939.475085] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8939.634875] LustreError: 443087:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8939.642214] LustreError: 443087:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8939.660236] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8940.944384] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 8943.688806] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8943.808781] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8945.059932] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 8947.569646] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8949.392590] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8951.625122] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8953.643622] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 8956.109769] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8966.053689] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 8974.815841] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8974.818886] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8974.824690] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8974.827699] Lustre: Skipped 1 previous similar message [ 8979.607246] Lustre: server umount lustre-MDT0000 complete [ 8980.927607] LustreError: 444931:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788912104 with bad export cookie 1857386101790420402 [ 8980.930802] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8980.933709] LustreError: 444931:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8981.053092] Lustre: server umount lustre-MDT0001 complete [ 8992.726078] Lustre: server umount lustre-OST0000 complete [ 8997.826239] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8997.982387] LustreError: 447020:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8998.006963] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8998.009790] Lustre: Skipped 1 previous similar message [ 8999.525079] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 9002.505062] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9004.005724] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 9006.364132] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9008.139133] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9010.447317] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9012.466519] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 9014.895166] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9022.886132] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 9028.066021] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9028.066662] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9028.067352] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9028.067356] Lustre: Skipped 3 previous similar messages [ 9028.068787] LustreError: Skipped 1 previous similar message [ 9028.080393] Lustre: Skipped 3 previous similar messages [ 9032.848623] Lustre: server umount lustre-OST0000 complete [ 9038.304449] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9038.305273] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9038.309670] Lustre: Skipped 1 previous similar message [ 9038.311946] Lustre: Skipped 1 previous similar message [ 9040.339204] Lustre: server umount lustre-MDT0000 complete [ 9041.621318] LustreError: 449618:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788912165 with bad export cookie 1857386101790421410 [ 9041.624404] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9041.625481] LustreError: 449618:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9041.740625] Lustre: server umount lustre-MDT0001 complete [ 9045.458732] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 9046.469723] Key type lgssc unregistered [ 9046.600376] LNet: 450571:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9046.603774] LNetError: 450571:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9046.614344] LNet: Removed LNI 192.168.204.126@tcp [ 9046.932682] Key type .llcrypt unregistered [ 9046.933927] Key type ._llcrypt unregistered [ 9057.037485] Key type ._llcrypt registered [ 9057.038758] Key type .llcrypt registered [ 9057.076713] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 9057.538490] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9057.550409] alg: No test for adler32 (adler32-zlib) [ 9058.405295] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 9058.492127] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 9060.071390] Key type lgssc registered [ 9060.471654] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9063.026080] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9063.032301] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9063.036030] LustreError: 451988:0:(osd_handler.c:8554:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 9063.043968] LustreError: 451988:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 9063.046421] LustreError: 451988:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 9063.048565] LustreError: 451988:0:(tgt_mount.c:2671:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 9063.051238] LustreError: 451988:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 9065.562409] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9065.565556] LustreError: 452385:0:(osd_handler.c:8554:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 9065.574263] LustreError: 452385:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [ 9065.576964] LustreError: 452385:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 9065.579672] LustreError: 452385:0:(tgt_mount.c:2671:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 9065.582790] LustreError: 452385:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 9071.838487] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 9072.986652] Key type lgssc unregistered [ 9073.128376] LNet: 453387:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9073.131277] LNetError: 453387:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9073.143299] LNet: Removed LNI 192.168.204.126@tcp [ 9073.472424] Key type .llcrypt unregistered [ 9073.474024] Key type ._llcrypt unregistered [ 9079.581021] Key type ._llcrypt registered [ 9079.582607] Key type .llcrypt registered [ 9079.623092] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [ 9085.746569] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 9086.088685] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9086.129144] alg: No test for adler32 (adler32-zlib) [ 9086.988299] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 9087.080300] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 9088.655152] Key type lgssc registered [ 9089.006574] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9099.676146] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9099.680910] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9100.763696] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9100.773073] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9100.807362] Lustre: lustre-MDT0000: new disk, initializing [ 9100.826052] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9100.830960] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9102.052892] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 9105.931455] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9105.970088] Lustre: 456513:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9106.003769] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9106.006394] Lustre: Skipped 1 previous similar message [ 9106.040131] Lustre: lustre-MDT0001: new disk, initializing [ 9106.071255] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9106.082943] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9106.087159] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9107.372938] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 9110.970740] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9112.779495] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9115.002402] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9115.097251] Lustre: lustre-OST0000: new disk, initializing [ 9115.099458] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9115.102635] Lustre: 458484:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9115.130482] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9116.880959] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 9120.250735] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9120.254791] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9120.292326] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9120.400818] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9121.789608] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9121.862665] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9123.149653] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9123.218023] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9125.346475] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9125.346953] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9125.350330] LustreError: Skipped 1 previous similar message [ 9125.354285] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9125.357018] Lustre: Skipped 1 previous similar message [ 9129.938952] Lustre: server umount lustre-OST0000 complete [ 9131.999818] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9132.001935] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9132.005011] Lustre: Skipped 1 previous similar message [ 9132.006568] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9135.584953] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9135.585305] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9140.704648] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9140.706485] Lustre: Skipped 2 previous similar messages [ 9145.311171] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9145.382781] Lustre: server umount lustre-MDT0000 complete [ 9145.824212] LustreError: 457332:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9145.829526] LustreError: 457332:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9146.564076] LustreError: 456506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788912270 with bad export cookie 12756478739472016522 [ 9146.567136] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9146.567698] LustreError: 456506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9146.677932] Lustre: server umount lustre-MDT0001 complete [ 9149.552671] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 9150.123706] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 20:04:32 (1788912272) [ 9152.348809] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9152.515026] LustreError: 460940:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9152.540671] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9153.965423] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 9156.815615] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9156.928099] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9158.267593] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 9160.627591] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9162.310841] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9164.520941] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9166.553172] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 9168.942744] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9171.106869] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9171.165097] Lustre: lustre-OST0001: new disk, initializing [ 9171.167277] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9171.169576] Lustre: 463772:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9171.195933] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9171.197621] Lustre: Skipped 1 previous similar message [ 9173.095716] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 9176.061824] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9176.066099] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9176.094925] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9176.828244] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9179.683652] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9179.690834] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9179.694870] Lustre: Skipped 1 previous similar message [ 9184.700475] Lustre: server umount lustre-OST0001 complete [ 9184.739869] LustreError: 464413:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9184.745531] LustreError: 464413:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9189.858441] LustreError: 462759:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9189.863525] LustreError: 462759:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9194.980506] LustreError: 462761:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9194.985767] LustreError: 462761:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9197.535783] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9197.538389] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9197.541923] Lustre: Skipped 1 previous similar message [ 9197.544197] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9197.546730] Lustre: Skipped 1 previous similar message [ 9200.100264] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9205.219611] LustreError: 462762:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9205.224019] LustreError: 462762:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9210.847122] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9210.894957] Lustre: server umount lustre-OST0000 complete [ 9213.407782] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9213.411090] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9213.416278] Lustre: Skipped 1 previous similar message [ 9215.456208] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9215.458263] Lustre: Skipped 8 previous similar messages [ 9218.321934] Lustre: server umount lustre-MDT0000 complete [ 9219.649194] LustreError: 460922:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788912343 with bad export cookie 12756478739472017411 [ 9219.651567] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9219.654089] LustreError: 460922:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9225.838367] Lustre: server umount lustre-MDT0001 complete [ 9229.249836] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 9230.256620] Key type lgssc unregistered [ 9230.396090] LNet: 465757:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9230.398989] LNetError: 465757:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9230.406417] LNet: Removed LNI 192.168.204.126@tcp [ 9230.716482] Key type .llcrypt unregistered [ 9230.717861] Key type ._llcrypt unregistered [ 9248.317771] Key type ._llcrypt registered [ 9248.319151] Key type .llcrypt registered [ 9248.357560] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 9248.800694] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9248.856518] alg: No test for adler32 (adler32-zlib) [ 9249.707158] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 9249.798163] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 9251.383110] Key type lgssc registered [ 9251.748169] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9254.355483] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9254.362148] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9255.450662] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9255.458736] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9255.513418] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9256.772434] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 9259.378089] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9259.411269] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9259.421387] Lustre: 468530:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9259.507044] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9260.797950] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 9263.216954] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9264.985164] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9267.168701] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9267.169299] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9267.173029] Lustre: Skipped 1 previous similar message [ 9267.176578] Lustre: Skipped 1 previous similar message [ 9271.893111] Lustre: server umount lustre-MDT0000 complete [ 9273.089292] LustreError: 468520:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788912396 with bad export cookie 9565474915459140880 [ 9273.093270] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9273.093437] LustreError: 468520:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9273.208682] Lustre: server umount lustre-MDT0001 complete [ 9276.440462] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 20:06:39 (1788912399) [ 9278.513559] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9278.652377] LustreError: 471039:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9278.656427] LustreError: 471039:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9278.668196] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9279.855297] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 9282.508575] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9282.614669] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9283.834781] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 9286.109982] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9287.802884] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9289.928436] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9289.994163] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9290.057220] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9291.955029] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 9294.277508] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9297.830728] Lustre: Permanently deactivating lustre-OST0000 [ 9297.834906] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 9299.442462] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 9300.934054] LustreError: 473927:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [ 9300.938604] LustreError: 473927:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [ 9300.941325] LustreError: 473927:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 9301.517672] LustreError: 473975:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [ 9301.519933] LustreError: 473975:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 9308.168850] Lustre: server umount lustre-OST0000 complete [ 9309.001203] LustreError: 474226:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [ 9309.004191] LustreError: 474226:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 9313.247792] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9313.249669] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9313.253269] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9314.272033] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9314.272409] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9319.392253] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9319.395091] Lustre: Skipped 2 previous similar messages [ 9319.416218] Lustre: server umount lustre-MDT0000 complete [ 9320.570210] LustreError: 471021:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788912444 with bad export cookie 9565474915459141447 [ 9320.573142] LustreError: 471021:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9320.573833] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9331.222510] Lustre: server umount lustre-MDT0001 complete [ 9333.801647] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9334.976516] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 9335.815691] LustreError: 475383:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [ 9337.685880] Lustre: 475722:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.126@tcp [ 9338.328631] Lustre: server umount MGS complete [ 9341.933653] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9342.087338] LustreError: 476444:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9342.092168] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 9342.096959] Lustre: Skipped 1 previous similar message [ 9342.116324] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9343.408494] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 9346.085806] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9346.183186] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 9347.561275] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 9349.913685] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9351.717276] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9352.314613] Lustre: Permanently reactivating lustre-OST0000 [ 9352.319556] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 9352.322259] Lustre: Skipped 2 previous similar messages [ 9381.343141] LustreError: 478369:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [ 9384.032673] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9384.142970] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9384.146464] Lustre: Skipped 1 previous similar message [ 9385.943329] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 9388.210994] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9400.800278] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9400.803090] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9400.808278] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9401.825776] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9402.125555] Lustre: server umount lustre-OST0000 complete [ 9405.392539] Lustre: server umount lustre-MDT0000 complete [ 9406.575272] LustreError: 476426:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788912530 with bad export cookie 9565474915459142469 [ 9406.577063] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9406.578502] LustreError: 476426:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9406.582447] Lustre: 476436:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.204.126@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 9406.689059] Lustre: server umount lustre-MDT0001 complete [ 9410.187077] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 9411.121836] Key type lgssc unregistered [ 9411.245264] LNet: 481258:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9411.247358] LNetError: 481258:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9411.256265] LNet: Removed LNI 192.168.204.126@tcp [ 9411.582414] Key type .llcrypt unregistered [ 9411.583401] Key type ._llcrypt unregistered [ 9417.348336] Key type ._llcrypt registered [ 9417.349864] Key type .llcrypt registered [ 9417.395552] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [ 9423.618740] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 9423.926781] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9423.969285] alg: No test for adler32 (adler32-zlib) [ 9424.821389] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 9424.909231] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 9426.495100] Key type lgssc registered [ 9426.822382] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9437.821533] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9437.827807] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9438.907413] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9438.917641] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9438.955418] Lustre: lustre-MDT0000: new disk, initializing [ 9438.973458] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9438.978349] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9440.242887] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 9444.269658] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9444.310056] Lustre: 484387:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9444.351567] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9444.353769] Lustre: Skipped 1 previous similar message [ 9444.391936] Lustre: lustre-MDT0001: new disk, initializing [ 9444.422978] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9444.437403] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9444.441123] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9445.687038] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 9449.306281] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9451.183819] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9453.422052] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9453.515628] Lustre: lustre-OST0000: new disk, initializing [ 9453.517872] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9453.521822] Lustre: 486352:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9453.549437] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9455.474723] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 9459.129492] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9460.533465] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9461.748586] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9461.751615] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9461.782810] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9462.644034] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 9463.958996] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9464.035647] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9466.849501] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9466.852717] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9466.855059] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9466.858051] Lustre: Skipped 1 previous similar message [ 9470.739290] Lustre: server umount lustre-OST0000 complete [ 9471.968132] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9471.968480] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9471.972377] Lustre: Skipped 1 previous similar message [ 9471.974425] Lustre: Skipped 1 previous similar message [ 9477.087943] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9477.089524] Lustre: Skipped 2 previous similar messages [ 9482.208239] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9482.210182] Lustre: Skipped 1 previous similar message [ 9486.303136] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9486.304551] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9486.307381] Lustre: Skipped 1 previous similar message [ 9486.364989] Lustre: server umount lustre-MDT0000 complete [ 9487.517089] LustreError: 484377:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788912611 with bad export cookie 2110555277288232445 [ 9487.519637] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9487.521740] LustreError: 484377:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9487.621070] Lustre: server umount lustre-MDT0001 complete [ 9490.224284] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [ 9490.752804] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 20:10:13 (1788912613) [ 9492.932063] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9493.096411] LustreError: 488823:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9493.101260] LustreError: 488823:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9493.120369] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9494.321192] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 9496.968461] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9498.378684] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 9500.659501] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9502.408864] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9504.587863] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9504.708324] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9504.710684] Lustre: Skipped 1 previous similar message [ 9506.459486] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 9508.797368] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9509.453959] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [ 9520.233648] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [ 9522.143874] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9522.147127] LustreError: Skipped 1 previous similar message [ 9522.148562] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9522.152918] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9523.169504] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9527.955449] Lustre: server umount lustre-OST0000 complete [ 9532.895833] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9532.899158] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9532.904395] Lustre: Skipped 1 previous similar message [ 9535.316690] Lustre: server umount lustre-MDT0000 complete [ 9536.529426] LustreError: 489556:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788912660 with bad export cookie 2110555277288233327 [ 9536.532514] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9536.533941] LustreError: 489556:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9536.639765] Lustre: server umount lustre-MDT0001 complete [ 9539.929320] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 9540.901783] Key type lgssc unregistered [ 9541.029527] LNet: 492565:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9541.032546] LNetError: 492565:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9541.041488] LNet: Removed LNI 192.168.204.126@tcp [ 9541.375540] Key type .llcrypt unregistered [ 9541.377105] Key type ._llcrypt unregistered [ 9548.743659] Key type ._llcrypt registered [ 9548.744731] Key type .llcrypt registered [ 9548.789285] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 9548.896440] Key type .llcrypt unregistered [ 9548.897849] Key type ._llcrypt unregistered [ 9555.712170] Key type ._llcrypt registered [ 9555.713364] Key type .llcrypt registered [ 9555.755449] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 9556.118576] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9556.152044] alg: No test for adler32 (adler32-zlib) [ 9556.994476] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 9557.077731] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 9558.655206] Key type lgssc registered [ 9558.970607] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9561.441779] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9561.446892] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9562.583693] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9563.868815] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 9566.502267] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9566.612880] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9568.510049] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 9570.785350] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9571.809770] LustreError: 494757:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9571.813286] LustreError: 494757:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 9572.834241] LustreError: 494776:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9572.839963] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 9572.840071] LustreError: 494776:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9573.021341] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9573.157379] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9574.513892] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 9583.071850] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9583.074042] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9583.078921] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9583.585233] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9583.589682] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9587.789270] Lustre: server umount lustre-OST0000 complete [ 9592.801404] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 9592.804309] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9592.809594] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9595.088691] Lustre: server umount lustre-MDT0000 complete [ 9596.321362] LustreError: 494739:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788912720 with bad export cookie 15546623346933287513 [ 9596.324531] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9596.325721] LustreError: 494739:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9596.446703] Lustre: server umount lustre-MDT0001 complete [ 9600.276338] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 9601.412139] Key type lgssc unregistered [ 9601.534413] LNet: 497904:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9601.536562] LNetError: 497904:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9601.543290] LNet: Removed LNI 192.168.204.126@tcp [ 9601.862044] Key type .llcrypt unregistered [ 9601.863067] Key type ._llcrypt unregistered [ 9611.038659] Key type ._llcrypt registered [ 9611.039843] Key type .llcrypt registered [ 9611.072338] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 9611.369180] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9611.413577] alg: No test for adler32 (adler32-zlib) [ 9612.279445] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 9612.371460] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 9613.951158] Key type lgssc registered [ 9614.302939] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9616.934465] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9616.941653] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9632.351409] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 9637.471299] LustreError: 499271:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.126@tcp: failed processing log, type 4: rc = -110 [ 9664.095197] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9666.035373] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 9668.549267] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9670.841259] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9671.042046] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9671.083516] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 9672.651602] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 9676.257173] LustreError: 499296:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9676.263719] LustreError: 499296:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9676.430333] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9676.617913] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9678.534519] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 9683.462333] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9687.786453] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9694.178862] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9694.188733] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9694.193239] Lustre: Skipped 1 previous similar message [ 9698.424818] Lustre: server umount lustre-OST0000 complete [ 9702.369110] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9702.377692] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9702.396695] Lustre: Skipped 1 previous similar message [ 9702.399849] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9704.417066] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9704.421217] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9707.461357] Lustre: server umount lustre-MDT0000 complete [ 9709.537220] LustreError: 500307:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9709.562626] LustreError: 500307:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9710.606736] LustreError: 502087:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788912834 with bad export cookie 11972857138539487493 [ 9710.610258] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9710.616767] LustreError: 502087:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9710.828266] Lustre: server umount lustre-MDT0001 complete [ 9717.754700] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 9719.105304] Key type lgssc unregistered [ 9719.252073] LNet: 502838:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9719.256605] LNetError: 502838:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9719.266925] LNet: Removed LNI 192.168.204.126@tcp [ 9719.728876] Key type .llcrypt unregistered [ 9719.730683] Key type ._llcrypt unregistered [ 9735.221810] Key type ._llcrypt registered [ 9735.223560] Key type .llcrypt registered [ 9735.313489] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 9735.920517] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9735.990076] alg: No test for adler32 (adler32-zlib) [ 9736.938532] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 9737.110098] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 9738.727269] Key type lgssc registered [ 9739.386036] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9743.612701] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9743.627103] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9744.867244] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9746.789282] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 9750.623878] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9750.852854] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9752.735406] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 9755.934413] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9758.350574] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9761.413515] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9761.588173] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9764.229431] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 9767.376897] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9769.774051] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 9776.608265] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9776.612203] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9776.617252] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9780.266077] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.26@tcp (stopping) [ 9781.219247] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9781.226090] Lustre: Skipped 1 previous similar message [ 9782.259438] Lustre: server umount lustre-MDT0000 complete [ 9783.374247] LustreError: 504213:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.26@tcp arrived at 1788912907 with bad export cookie 12966340468268843336 [ 9783.381481] LustreError: 504213:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9784.320186] LustreError: 506923:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788912908 with bad export cookie 12966340468268842720 [ 9784.325698] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9784.326158] LustreError: 506923:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9786.336428] LustreError: 504971:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9786.337744] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9786.338319] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9786.339158] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9786.339161] Lustre: Skipped 3 previous similar messages [ 9786.344429] LustreError: 504971:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9789.344531] LustreError: 504232:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9789.345562] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9789.352070] LustreError: 504232:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9790.447483] Lustre: server umount lustre-OST0000 complete [ 9793.642435] Lustre: server umount lustre-MDT0001 complete [ 9798.120917] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 9799.402339] Key type lgssc unregistered [ 9799.558458] LNet: 507912:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9799.567543] LNetError: 507912:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9799.575474] LNet: Removed LNI 192.168.204.126@tcp [ 9799.927880] Key type .llcrypt unregistered [ 9799.929279] Key type ._llcrypt unregistered [ 9810.626533] Key type ._llcrypt registered [ 9810.628170] Key type .llcrypt registered [ 9810.675856] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 9811.134459] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9811.144238] alg: No test for adler32 (adler32-zlib) [ 9812.007525] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 9812.111427] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 9813.703191] Key type lgssc registered [ 9814.222661] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9817.546763] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9817.557832] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9818.719771] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9820.376766] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 9823.968887] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9824.128144] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9825.850954] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 9828.928513] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9831.265942] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9834.289262] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9834.481139] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9837.206748] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 9840.287636] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9843.688652] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 9848.804049] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 9848.805340] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9848.818281] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9848.822033] Lustre: Skipped 1 previous similar message [ 9852.950331] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.26@tcp (stopping) [ 9854.873775] Lustre: server umount lustre-MDT0001 complete [ 9859.041061] LustreError: 509305:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9859.042043] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9859.048277] LustreError: 509305:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9859.055952] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9859.062589] Lustre: Skipped 1 previous similar message [ 9859.066597] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9859.069266] Lustre: Skipped 2 previous similar messages [ 9863.191209] Lustre: server umount lustre-OST0000 complete [ 9864.160572] LustreError: 510046:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9864.715759] Lustre: server umount lustre-MDT0000 complete [ 9870.008170] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 9871.111085] Key type lgssc unregistered [ 9871.238776] LNet: 512989:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9871.241643] LNetError: 512989:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9871.251475] LNet: Removed LNI 192.168.204.126@tcp [ 9871.580412] Key type .llcrypt unregistered [ 9871.581482] Key type ._llcrypt unregistered [ 9880.343921] Key type ._llcrypt registered [ 9880.345275] Key type .llcrypt registered [ 9880.385022] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [ 9887.065877] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 9887.384291] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9887.417029] alg: No test for adler32 (adler32-zlib) [ 9888.263964] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 9888.350101] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 9889.927183] Key type lgssc registered [ 9890.346309] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9904.483159] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 9910.871783] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9910.878829] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9911.976564] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9911.990105] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9912.028992] Lustre: lustre-MDT0000: new disk, initializing [ 9912.057208] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9912.065395] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9913.647770] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 9920.216071] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9920.327222] Lustre: lustre-OST0000: new disk, initializing [ 9920.329594] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9920.332165] Lustre: Skipped 1 previous similar message [ 9920.334874] Lustre: 517996:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9920.366118] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9923.601474] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 9925.214555] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 9925.227867] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 9925.255906] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 9932.671299] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9945.788675] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9945.878480] Lustre: 516976:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9945.943048] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9946.022551] Lustre: lustre-MDT0001: new disk, initializing [ 9946.145862] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9946.198439] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 9946.209315] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 9950.245877] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 9959.690293] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 9959.960209] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 9964.535898] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 9964.714217] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 9972.209126] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9972.213837] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 9972.227165] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9973.805172] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.26@tcp (stopping) [ 9973.812020] Lustre: Skipped 1 previous similar message [ 9975.870120] Lustre: server umount lustre-MDT0001 complete [ 9982.440944] LustreError: 516987:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9982.464901] LustreError: 516987:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 9987.557197] LustreError: 516984:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9987.566603] LustreError: 516984:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9992.675308] LustreError: 516988:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9992.693630] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9992.712727] Lustre: Skipped 1 previous similar message [ 9992.717363] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9997.795131] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9997.796159] LustreError: 516987:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9998.015550] Lustre: server umount lustre-OST0000 complete [10006.711241] Lustre: server umount lustre-MDT0000 complete [10013.360527] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [10014.654385] Key type lgssc unregistered [10014.809880] LNet: 521742:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10014.814528] LNetError: 521742:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10014.822566] LNet: Removed LNI 192.168.204.126@tcp [10015.181850] Key type .llcrypt unregistered [10015.183389] Key type ._llcrypt unregistered [10035.557273] Key type ._llcrypt registered [10035.558882] Key type .llcrypt registered [10035.620942] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [10035.782472] Key type .llcrypt unregistered [10035.784798] Key type ._llcrypt unregistered [10046.108120] Key type ._llcrypt registered [10046.110563] Key type .llcrypt registered [10046.174875] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [10046.854426] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10046.873343] alg: No test for adler32 (adler32-zlib) [10047.856542] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [10047.995756] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [10049.615479] Key type lgssc registered [10050.290882] Lustre: Echo OBD driver; http://www.lustre.org/ [10065.120780] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10065.140915] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10066.328717] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10066.348367] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10066.402989] Lustre: lustre-MDT0000: new disk, initializing [10066.446264] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10066.461532] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10068.708026] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [10074.685119] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10074.738232] Lustre: 525092:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10074.767869] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10074.770882] Lustre: Skipped 1 previous similar message [10074.815601] Lustre: lustre-MDT0001: new disk, initializing [10074.859428] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10074.877422] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10074.881379] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10077.012124] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [10082.009320] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10084.750823] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10088.571744] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10088.711918] Lustre: lustre-OST0000: new disk, initializing [10088.715126] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10088.720964] Lustre: 527061:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10088.785268] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10092.403602] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [10097.669600] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10097.676247] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10097.744855] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10098.101659] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10102.754192] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10102.757472] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10102.762419] Lustre: Skipped 2 previous similar messages [10102.769441] Lustre: Skipped 1 previous similar message [10107.919986] Lustre: server umount lustre-MDT0000 complete [10110.275104] LustreError: 525889:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788913234 with bad export cookie 10515339414957365286 [10110.283423] LustreError: 525889:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10110.289890] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10110.460364] Lustre: server umount lustre-MDT0001 complete [10122.872344] Lustre: server umount lustre-OST0000 complete [10128.424500] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 20:20:50 (1788913250) [10135.465660] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10135.935269] LustreError: 529086:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10135.960444] LustreError: 529086:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [10136.015861] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10141.152817] LustreError: 529087:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10142.737944] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [10145.250136] LustreError: 529086:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10150.394153] LustreError: 529087:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10154.113784] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10154.719047] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10162.139733] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [10173.568197] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10180.990236] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10200.486263] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10200.645195] Lustre: Found index 0 for lustre-OST0000, updating log [10200.784920] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10205.959579] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [10206.211583] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [10212.853711] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10221.540385] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10221.541257] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10221.552508] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10222.633476] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10222.946618] Lustre: server umount lustre-OST0000 complete [10240.932582] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10240.953587] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10247.137628] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10247.142948] Lustre: Skipped 3 previous similar messages [10252.256961] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10252.379763] Lustre: server umount lustre-MDT0000 complete [10255.392699] LustreError: 531932:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788913379 with bad export cookie 10515339414957366679 [10255.397437] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10255.405283] LustreError: 531932:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10255.730431] Lustre: server umount lustre-MDT0001 complete [10260.002239] LNet: 1 local NIs in recovery (showing 1): 192.168.204.126@tcp [10262.373428] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 20:23:04 (1788913384) [10860.967379] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10861.177478] LustreError: 535495:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10861.211082] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10862.915534] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [10866.496743] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10866.717825] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10868.603271] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [10871.895450] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10874.463481] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10890.666547] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10890.801221] Lustre: Found index 0 for lustre-OST0000, updating log [10890.943074] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10893.509371] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [10896.779319] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10899.426551] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10899.432892] Lustre: Skipped 1 previous similar message [10899.435074] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10899.436054] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10899.441104] Lustre: Skipped 2 previous similar messages [10903.829572] Lustre: server umount lustre-OST0000 complete [10906.467323] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10906.475623] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10906.479320] Lustre: Skipped 1 previous similar message [10908.512479] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10908.516320] Lustre: Skipped 3 previous similar messages [10911.585506] LustreError: 535496:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10911.591501] Lustre: server umount lustre-MDT0000 complete [10911.593244] LustreError: 535496:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10912.288387] LNet: 1 local NIs in recovery (showing 1): 192.168.204.126@tcp [10912.612648] LustreError: 535490:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10912.621301] LustreError: 535490:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10913.334807] LustreError: 535475:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788914037 with bad export cookie 10515339414957367750 [10913.337388] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10913.342686] LustreError: 535475:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10913.487640] Lustre: server umount lustre-MDT0001 complete [10932.914786] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [10937.082411] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [10952.555517] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 20:34:35 (1788914075) [10959.808081] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [10961.257953] Key type lgssc unregistered [10964.511491] LNet: 542108:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10964.514491] LNetError: 542108:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11019.817878] LNet: Removed LNI 192.168.204.126@tcp [11020.798171] Key type .llcrypt unregistered [11020.802103] Key type ._llcrypt unregistered [11048.295930] Key type ._llcrypt registered [11048.297453] Key type .llcrypt registered [11048.499617] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [11050.901481] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11051.101800] alg: No test for adler32 (adler32-zlib) [11052.211379] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [11052.518979] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [11054.303179] Key type lgssc registered [11056.153780] Lustre: Echo OBD driver; http://www.lustre.org/ [11061.783873] LNet: Added LNI 192.168.204.126@tcp100 [8/256/0/180] [11063.476429] LNet: Added LNI 192.168.204.126@tcp101 [8/256/0/180] [11064.839269] LNet: Added LNI 192.168.204.126@tcp102 [8/256/0/180] [11066.042062] LNet: Added LNI 192.168.204.126@tcp103 [8/256/0/180] [11067.241896] LNet: Added LNI 192.168.204.126@tcp104 [8/256/0/180] [11068.649143] LNet: Added LNI 192.168.204.126@tcp105 [8/256/0/180] [11070.110115] LNet: Added LNI 192.168.204.126@tcp106 [8/256/0/180] [11072.241936] LNet: Added LNI 192.168.204.126@tcp107 [8/256/0/180] [11073.981849] LNet: Added LNI 192.168.204.126@tcp108 [8/256/0/180] [11075.529376] LNet: Added LNI 192.168.204.126@tcp109 [8/256/0/180] [11077.167564] LNet: Added LNI 192.168.204.126@tcp110 [8/256/0/180] [11078.389644] LNet: Added LNI 192.168.204.126@tcp111 [8/256/0/180] [11079.276881] LNet: Added LNI 192.168.204.126@tcp112 [8/256/0/180] [11080.546752] LNet: Added LNI 192.168.204.126@tcp113 [8/256/0/180] [11081.858282] LNet: Added LNI 192.168.204.126@tcp114 [8/256/0/180] [11083.427866] LNet: Added LNI 192.168.204.126@tcp115 [8/256/0/180] [11085.057706] LNet: Added LNI 192.168.204.126@tcp116 [8/256/0/180] [11086.547201] LNet: Added LNI 192.168.204.126@tcp117 [8/256/0/180] [11088.089881] LNet: Added LNI 192.168.204.126@tcp118 [8/256/0/180] [11089.461939] LNet: Added LNI 192.168.204.126@tcp119 [8/256/0/180] [11090.529325] LNet: Added LNI 192.168.204.126@tcp120 [8/256/0/180] [11091.357165] LNet: Added LNI 192.168.204.126@tcp121 [8/256/0/180] [11092.396736] LNet: Added LNI 192.168.204.126@tcp122 [8/256/0/180] [11093.488821] LNet: Added LNI 192.168.204.126@tcp123 [8/256/0/180] [11094.509343] LNet: Added LNI 192.168.204.126@tcp124 [8/256/0/180] [11095.319693] LNet: Added LNI 192.168.204.126@tcp125 [8/256/0/180] [11096.087723] LNet: Added LNI 192.168.204.126@tcp126 [8/256/0/180] [11096.945651] LNet: Added LNI 192.168.204.126@tcp127 [8/256/0/180] [11097.897875] LNet: Added LNI 192.168.204.126@tcp128 [8/256/0/180] [11098.945464] LNet: Added LNI 192.168.204.126@tcp129 [8/256/0/180] [11099.948171] LNet: Added LNI 192.168.204.126@tcp130 [8/256/0/180] [11100.938091] LNet: Added LNI 192.168.204.126@tcp131 [8/256/0/180] [11101.970659] LNet: Added LNI 192.168.204.126@tcp132 [8/256/0/180] [11102.946373] LNet: Added LNI 192.168.204.126@tcp133 [8/256/0/180] [11104.100592] LNet: Added LNI 192.168.204.126@tcp134 [8/256/0/180] [11104.908258] LNet: Added LNI 192.168.204.126@tcp135 [8/256/0/180] [11111.984139] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11112.026113] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11113.218943] Lustre: 545139:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.126@tcp: can't add NID 192.168.204.126@tcp130: rc = -75 [11113.359086] LNetError: 545160:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.126 [11113.399781] LNetError: Protocol error connecting to 192.168.204.126@tcp119 at host 192.168.204.126:988: is it running a compatible version of Lustre? [11113.424465] LNetError: Skipped 6 previous similar messages [11113.676448] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11113.723459] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11113.951184] LNet: 3 local NIs in recovery (showing 3): 192.168.204.126@tcp123, 192.168.204.126@tcp128, 192.168.204.126@tcp119 [11113.963781] Lustre: lustre-MDT0000: new disk, initializing [11114.149105] Lustre: 545196:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.126@tcp: can't add NID 192.168.204.126@tcp130: rc = -75 [11114.157456] Lustre: 545196:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [11114.174973] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11114.190785] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11114.464583] LNetError: 545163:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.126 [11114.487822] LNetError: 545163:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 7 previous similar messages [11114.506416] LNetError: Protocol error connecting to 192.168.204.126@tcp124 at host 192.168.204.126:988: is it running a compatible version of Lustre? [11114.529396] LNetError: Skipped 1 previous similar message [11117.039844] LNetError: 545171:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.126 [11117.052917] LNetError: 545171:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 3 previous similar messages [11117.065201] LNetError: Protocol error connecting to 192.168.204.126@tcp123 at host 192.168.204.126:988: is it running a compatible version of Lustre? [11117.075268] LNetError: Skipped 5 previous similar messages [11119.946862] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [11121.132283] LNetError: 545167:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.126 [11121.150909] LNetError: 545167:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 2 previous similar messages [11121.163184] LNetError: Protocol error connecting to 192.168.204.126@tcp123 at host 192.168.204.126:988: is it running a compatible version of Lustre? [11129.323280] LNetError: 545163:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.126 [11129.346421] LNetError: Protocol error connecting to 192.168.204.126@tcp123 at host 192.168.204.126:988: is it running a compatible version of Lustre? [11135.192630] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11135.456738] Lustre: 545156:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11135.630691] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11135.645459] Lustre: Skipped 1 previous similar message [11135.786629] Lustre: lustre-MDT0001: new disk, initializing [11135.849463] Lustre: 545959:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.126@tcp: can't add NID 192.168.204.126@tcp130: rc = -75 [11135.859632] Lustre: 545959:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [11136.041709] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11136.133793] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11136.147710] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11141.123958] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [11143.236775] Lustre: 546324:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.126@tcp: can't add NID 192.168.204.126@tcp130: rc = -75 [11143.260371] Lustre: 546324:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [11143.648884] LNetError: 542597:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.126 [11143.659238] LNetError: Protocol error connecting to 192.168.204.126@tcp124 at host 192.168.204.126:988: is it running a compatible version of Lustre? [11143.667595] LNetError: Skipped 1 previous similar message [11151.668160] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11157.848481] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11161.060861] LNetError: 542596:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.126 [11161.100240] LNetError: 542596:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 10 previous similar messages [11161.134773] LNetError: Protocol error connecting to 192.168.204.126@tcp125 at host 192.168.204.126:988: is it running a compatible version of Lustre? [11161.158842] LNetError: Skipped 9 previous similar messages [11166.201126] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11166.713398] Lustre: lustre-OST0000: new disk, initializing [11166.721929] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11166.728962] Lustre: 547132:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11166.865478] Lustre: 547137:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.126@tcp: can't add NID 192.168.204.126@tcp130: rc = -75 [11166.875949] Lustre: 547137:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [11166.939096] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11173.343837] LNet: 1 local NIs in recovery (showing 1): 192.168.204.126@tcp123 [11174.067420] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [11176.452950] Lustre: 547589:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.126@tcp: can't add NID 192.168.204.126@tcp130: rc = -75 [11176.460265] Lustre: 547589:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [11176.523363] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11176.535293] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11176.641773] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11183.604618] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11192.292028] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11192.299527] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11192.316462] LustreError: Skipped 1 previous similar message [11192.318599] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11192.360819] Lustre: Skipped 1 previous similar message [11194.850405] LNetError: 542595:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.126 [11194.873855] LNetError: 542595:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 16 previous similar messages [11194.888912] LNetError: Protocol error connecting to 192.168.204.126@tcp124 at host 192.168.204.126:988: is it running a compatible version of Lustre? [11194.908716] LNetError: Skipped 16 previous similar messages [11197.471236] Lustre: server umount lustre-OST0000 complete [11202.529383] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11202.532955] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11202.547177] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11202.552203] Lustre: Skipped 1 previous similar message [11202.562929] Lustre: Skipped 2 previous similar messages [11207.652291] LustreError: 545174:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11207.661797] LustreError: 545174:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [11207.738706] Lustre: server umount lustre-MDT0000 complete [11211.311931] LustreError: 545149:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788914335 with bad export cookie 7092236586058155301 [11211.328213] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11211.634390] Lustre: server umount lustre-MDT0001 complete [11223.424890] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [11226.417926] Key type lgssc unregistered [11226.782072] LNet: 549046:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11226.807127] LNetError: 549046:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11226.820353] LNet: Removed LNI 192.168.204.126@tcp [11227.875226] LNet: Removed LNI 192.168.204.126@tcp100 [11227.879257] LNetError: 549046:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11227.897311] LNetError: 549046:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [11227.908114] LNet: Removed LNI 192.168.204.126@tcp101 [11228.959519] LNet: Removed LNI 192.168.204.126@tcp102 [11228.965792] LNetError: 549046:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11228.972658] LNetError: 549046:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [11228.983703] LNet: Removed LNI 192.168.204.126@tcp103 [11230.050576] LNet: Removed LNI 192.168.204.126@tcp104 [11231.074319] LNet: Removed LNI 192.168.204.126@tcp105 [11231.077533] LNetError: 549046:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11231.087844] LNetError: 549046:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [11231.097609] LNet: Removed LNI 192.168.204.126@tcp106 [11232.159235] LNet: Removed LNI 192.168.204.126@tcp107 [11233.183177] LNet: Removed LNI 192.168.204.126@tcp108 [11234.273347] LNet: Removed LNI 192.168.204.126@tcp109 [11235.295387] LNet: Removed LNI 192.168.204.126@tcp110 [11235.303912] LNetError: 549046:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11235.314794] LNetError: 549046:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [11235.322924] LNet: Removed LNI 192.168.204.126@tcp111 [11236.385347] LNet: Removed LNI 192.168.204.126@tcp112 [11237.409769] LNet: Removed LNI 192.168.204.126@tcp113 [11238.431182] LNet: Removed LNI 192.168.204.126@tcp114 [11239.455343] LNet: Removed LNI 192.168.204.126@tcp115 [11240.479173] LNet: Removed LNI 192.168.204.126@tcp116 [11241.503272] LNet: Removed LNI 192.168.204.126@tcp117 [11242.527173] LNet: Removed LNI 192.168.204.126@tcp118 [11243.551233] LNet: Removed LNI 192.168.204.126@tcp119 [11243.557909] LNetError: 549046:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11243.566923] LNetError: 549046:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [11243.578421] LNet: Removed LNI 192.168.204.126@tcp120 [11244.639222] LNet: Removed LNI 192.168.204.126@tcp121 [11245.666411] LNet: Removed LNI 192.168.204.126@tcp122 [11245.672625] LNet: Removed LNI 192.168.204.126@tcp123 [11246.687221] LNet: Removed LNI 192.168.204.126@tcp124 [11247.711772] LNet: Removed LNI 192.168.204.126@tcp125 [11248.735195] LNet: Removed LNI 192.168.204.126@tcp126 [11249.759139] LNet: Removed LNI 192.168.204.126@tcp127 [11250.785945] LNet: Removed LNI 192.168.204.126@tcp128 [11251.807314] LNet: Removed LNI 192.168.204.126@tcp129 [11251.813074] LNet: Removed LNI 192.168.204.126@tcp130 [11251.823609] LNet: Removed LNI 192.168.204.126@tcp131 [11251.832898] LNet: Removed LNI 192.168.204.126@tcp132 [11251.840974] LNet: Removed LNI 192.168.204.126@tcp133 [11251.847879] LNet: Removed LNI 192.168.204.126@tcp134 [11251.862613] LNet: Removed LNI 192.168.204.126@tcp135 [11252.627345] Key type .llcrypt unregistered [11252.629334] Key type ._llcrypt unregistered [11278.265569] Key type ._llcrypt registered [11278.270312] Key type .llcrypt registered [11278.438198] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [11294.879191] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [11296.362923] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11296.426540] alg: No test for adler32 (adler32-zlib) [11297.605177] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [11297.978844] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [11299.769253] Key type lgssc registered [11301.209202] Lustre: Echo OBD driver; http://www.lustre.org/ [11343.426507] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 20:41:04 (1788914464) [11374.748640] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [11378.080282] Key type lgssc unregistered [11378.507749] LNet: 553157:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11378.522721] LNetError: 553157:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11378.556343] LNet: Removed LNI 192.168.204.126@tcp [11380.069526] Key type .llcrypt unregistered [11380.073175] Key type ._llcrypt unregistered [11399.285839] Key type ._llcrypt registered [11399.289399] Key type .llcrypt registered [11399.378554] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [11400.919509] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11401.001490] alg: No test for adler32 (adler32-zlib) [11402.293565] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [11402.723237] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [11404.471186] Key type lgssc registered [11406.243790] Lustre: Echo OBD driver; http://www.lustre.org/ [11421.952742] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11422.021957] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11423.599061] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11423.663380] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11423.878467] Lustre: lustre-MDT0000: new disk, initializing [11424.106265] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11424.160360] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11434.244848] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [11452.838248] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11453.083709] Lustre: 554373:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11453.203877] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11453.207741] Lustre: Skipped 1 previous similar message [11453.354742] Lustre: lustre-MDT0001: new disk, initializing [11453.572701] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11453.622131] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11453.639908] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11459.783924] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [11471.519200] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11481.542516] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11492.247123] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11492.670302] Lustre: lustre-OST0000: new disk, initializing [11492.675984] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11492.686944] Lustre: 556345:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11492.843908] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11498.973562] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [11502.668704] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11502.691563] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11502.876592] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11509.624603] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11513.385800] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing lnet_if_list [11519.342892] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [11519.367365] LNet: Added LNI 192.168.204.126@tcp10 [8/256/0/180] [11526.079508] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [11526.115408] LNet: Added LNI 192.168.204.126@tcp11 [8/256/0/180] [11530.742788] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [11530.780575] LNet: Added LNI 192.168.204.126@tcp12 [8/256/0/180] [11535.806150] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [11535.842244] LNet: Added LNI 192.168.204.126@tcp13 [8/256/0/180] [11541.664324] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [11541.701912] LNet: Added LNI 192.168.204.126@tcp14 [8/256/0/180] [11547.358113] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [11547.413559] LNet: Added LNI 192.168.204.126@tcp15 [8/256/0/180] [11552.351549] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [11552.375597] LNet: Added LNI 192.168.204.126@tcp16 [8/256/0/180] [11557.314750] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [11557.340125] LNet: Added LNI 192.168.204.126@tcp17 [8/256/0/180] [11562.322181] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [11562.354827] LNet: Added LNI 192.168.204.126@tcp18 [8/256/0/180] [11566.964497] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [11566.990038] LNet: Added LNI 192.168.204.126@tcp19 [8/256/0/180] [11572.207445] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [11572.229318] LNet: Added LNI 192.168.204.126@tcp20 [8/256/0/180] [11578.051892] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [11578.093860] LNet: Added LNI 192.168.204.126@tcp21 [8/256/0/180] [11583.559523] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [11583.594438] LNet: Added LNI 192.168.204.126@tcp22 [8/256/0/180] [11589.601517] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [11589.674877] LNet: Added LNI 192.168.204.126@tcp23 [8/256/0/180] [11596.122232] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [11596.146603] LNet: Added LNI 192.168.204.126@tcp24 [8/256/0/180] [11601.968366] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [11601.991820] LNet: Added LNI 192.168.204.126@tcp25 [8/256/0/180] [11607.487880] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [11607.550160] LNet: Added LNI 192.168.204.126@tcp26 [8/256/0/180] [11613.706531] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [11613.734160] LNet: Added LNI 192.168.204.126@tcp27 [8/256/0/180] [11620.320444] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [11620.358898] LNet: Added LNI 192.168.204.126@tcp28 [8/256/0/180] [11626.770345] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [11626.826271] LNet: Added LNI 192.168.204.126@tcp29 [8/256/0/180] [11633.915590] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [11633.948429] LNet: Added LNI 192.168.204.126@tcp30 [8/256/0/180] [11641.610045] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [11641.648523] LNet: Added LNI 192.168.204.126@tcp31 [8/256/0/180] [11648.820181] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [11648.873871] LNet: Added LNI 192.168.204.126@tcp32 [8/256/0/180] [11656.209669] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [11656.277983] LNet: Added LNI 192.168.204.126@tcp33 [8/256/0/180] [11663.377741] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [11663.405913] LNet: Added LNI 192.168.204.126@tcp34 [8/256/0/180] [11672.000180] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [11672.059320] LNet: Added LNI 192.168.204.126@tcp35 [8/256/0/180] [11680.819549] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [11680.847684] LNet: Added LNI 192.168.204.126@tcp36 [8/256/0/180] [11689.817855] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [11689.865237] LNet: Added LNI 192.168.204.126@tcp37 [8/256/0/180] [11697.900574] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [11697.936591] LNet: Added LNI 192.168.204.126@tcp38 [8/256/0/180] [11707.089656] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [11707.155702] LNet: Added LNI 192.168.204.126@tcp39 [8/256/0/180] [11717.354941] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [11717.415158] LNet: Added LNI 192.168.204.126@tcp40 [8/256/0/180] [11730.342649] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [11730.411688] LNet: Added LNI 192.168.204.126@tcp41 [8/256/0/180] [11743.157394] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [11743.203853] LNet: Added LNI 192.168.204.126@tcp42 [8/256/0/180] [11755.664652] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [11755.727360] LNet: Added LNI 192.168.204.126@tcp43 [8/256/0/180] [11765.065228] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [11765.103786] LNet: Added LNI 192.168.204.126@tcp44 [8/256/0/180] [11775.175982] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [11775.217206] LNet: Added LNI 192.168.204.126@tcp45 [8/256/0/180] [11784.555746] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [11784.579795] LNet: Added LNI 192.168.204.126@tcp46 [8/256/0/180] [11792.794241] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [11792.850230] LNet: Added LNI 192.168.204.126@tcp47 [8/256/0/180] [11800.403086] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [11800.424073] LNet: Added LNI 192.168.204.126@tcp48 [8/256/0/180] [11808.541707] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [11808.575588] LNet: Added LNI 192.168.204.126@tcp49 [8/256/0/180] [11817.740135] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [11817.770247] LNet: Added LNI 192.168.204.126@tcp50 [8/256/0/180] [11837.408851] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11837.421448] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11837.452157] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11838.959714] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11838.960210] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11838.981910] Lustre: Skipped 1 previous similar message [11841.618851] Lustre: server umount lustre-MDT0000 complete [11846.079977] LustreError: 554365:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788914969 with bad export cookie 16050060483760532171 [11846.094043] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11846.096196] LustreError: 554365:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11846.720333] Lustre: server umount lustre-MDT0001 complete [11865.569622] Lustre: 553646:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788914973/real 1788914973] req@000000008ec12c31 x1875812838201216/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788914989 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11865.611307] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11867.049538] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [11867.627398] Lustre: 553645:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788914975/real 1788914975] req@00000000197b15de x1875812838201344/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788914991 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11870.698883] Lustre: 553644:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788914978/real 1788914978] req@0000000092a58cb4 x1875812838201472/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788914994 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11876.760397] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11876.837089] Lustre: 553646:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788914984/real 1788914984] req@000000007587ea4a x1875812838201856/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788915000 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11876.878482] Lustre: 553646:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [11881.951275] Lustre: 553645:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788914989/real 1788914989] req@00000000061db597 x1875812838201984/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788915005 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11890.603073] LustreError: 556339:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11890.625855] LustreError: 556339:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [11890.682308] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11894.753454] LustreError: 556340:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11894.756307] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [11895.807212] LustreError: 556339:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11895.967873] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [11900.902961] LustreError: 556340:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11906.017219] LustreError: 556339:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11906.684147] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11907.250178] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11910.330021] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [11912.076924] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [11920.601343] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11926.354448] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11946.466076] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11946.468424] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11946.480596] Lustre: Skipped 1 previous similar message [11946.488231] Lustre: Skipped 2 previous similar messages [11951.609476] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11951.614588] Lustre: Skipped 1 previous similar message [11956.712847] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11956.725707] Lustre: Skipped 1 previous similar message [11960.287492] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11960.573905] Lustre: server umount lustre-OST0000 complete [11966.956080] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11966.958804] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11966.973135] Lustre: Skipped 1 previous similar message [11966.991021] Lustre: Skipped 1 previous similar message [11971.445525] Lustre: server umount lustre-MDT0000 complete [11972.065862] LustreError: 580600:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11972.111838] LustreError: 580600:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11976.539325] LustreError: 555172:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788915100 with bad export cookie 16050060483760534740 [11976.548475] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11976.551597] LustreError: 555172:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11977.254913] Lustre: server umount lustre-MDT0001 complete [11991.154504] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [11994.596746] Key type lgssc unregistered [11995.045625] LNet: 583389:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11995.055144] LNetError: 583389:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11995.063558] LNet: Removed LNI 192.168.204.126@tcp [11995.066706] LNet: Removed LNI 192.168.204.126@tcp10 [11995.070427] LNet: Removed LNI 192.168.204.126@tcp11 [11995.073407] LNet: Removed LNI 192.168.204.126@tcp12 [11995.076454] LNet: Removed LNI 192.168.204.126@tcp13 [11995.079871] LNet: Removed LNI 192.168.204.126@tcp14 [11995.083561] LNet: Removed LNI 192.168.204.126@tcp15 [11995.086746] LNet: Removed LNI 192.168.204.126@tcp16 [11995.089389] LNet: Removed LNI 192.168.204.126@tcp17 [11995.092133] LNet: Removed LNI 192.168.204.126@tcp18 [11995.095439] LNet: Removed LNI 192.168.204.126@tcp19 [11995.098948] LNet: Removed LNI 192.168.204.126@tcp20 [11995.102573] LNet: Removed LNI 192.168.204.126@tcp21 [11995.111097] LNet: Removed LNI 192.168.204.126@tcp22 [11995.120822] LNet: Removed LNI 192.168.204.126@tcp23 [11995.125379] LNet: Removed LNI 192.168.204.126@tcp24 [11995.128792] LNet: Removed LNI 192.168.204.126@tcp25 [11995.132150] LNet: Removed LNI 192.168.204.126@tcp26 [11995.136490] LNet: Removed LNI 192.168.204.126@tcp27 [11995.140466] LNet: Removed LNI 192.168.204.126@tcp28 [11995.143415] LNet: Removed LNI 192.168.204.126@tcp29 [11995.146754] LNet: Removed LNI 192.168.204.126@tcp30 [11995.150452] LNet: Removed LNI 192.168.204.126@tcp31 [11995.153631] LNet: Removed LNI 192.168.204.126@tcp32 [11995.156309] LNet: Removed LNI 192.168.204.126@tcp33 [11995.158793] LNet: Removed LNI 192.168.204.126@tcp34 [11995.161158] LNet: Removed LNI 192.168.204.126@tcp35 [11995.164429] LNet: Removed LNI 192.168.204.126@tcp36 [11995.167280] LNet: Removed LNI 192.168.204.126@tcp37 [11995.170174] LNet: Removed LNI 192.168.204.126@tcp38 [11995.172386] LNet: Removed LNI 192.168.204.126@tcp39 [11995.176172] LNet: Removed LNI 192.168.204.126@tcp40 [11995.182545] LNet: Removed LNI 192.168.204.126@tcp41 [11995.185447] LNet: Removed LNI 192.168.204.126@tcp42 [11995.189393] LNet: Removed LNI 192.168.204.126@tcp43 [11995.193873] LNet: Removed LNI 192.168.204.126@tcp44 [11995.200708] LNet: Removed LNI 192.168.204.126@tcp45 [11995.205217] LNet: Removed LNI 192.168.204.126@tcp46 [11995.210424] LNet: Removed LNI 192.168.204.126@tcp47 [11995.215984] LNet: Removed LNI 192.168.204.126@tcp48 [11995.218086] LNet: Removed LNI 192.168.204.126@tcp49 [11995.228647] LNet: Removed LNI 192.168.204.126@tcp50 [11996.564469] Key type .llcrypt unregistered [11996.566831] Key type ._llcrypt unregistered [12018.798317] Key type ._llcrypt registered [12018.800458] Key type .llcrypt registered [12018.891410] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [12036.162716] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [12037.662501] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12037.762829] alg: No test for adler32 (adler32-zlib) [12038.892425] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [12039.290908] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [12041.023257] Key type lgssc registered [12042.424779] Lustre: Echo OBD driver; http://www.lustre.org/ [12084.171424] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12084.242731] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12085.709301] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12085.751208] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12085.850330] Lustre: lustre-MDT0000: new disk, initializing [12085.942290] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12085.973922] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12091.986844] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [12105.048903] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12105.245585] Lustre: 586591:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12105.359881] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12105.372324] Lustre: Skipped 1 previous similar message [12105.540832] Lustre: lustre-MDT0001: new disk, initializing [12105.754357] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12105.864659] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12105.883763] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12111.133399] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [12123.285243] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12130.484368] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12138.995372] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12139.138345] Lustre: *** cfs_fail_loc=90f, val=0*** [12139.191871] Lustre: lustre-OST0000: new disk, initializing [12139.195372] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12139.200214] Lustre: 588609:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12139.283404] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12145.836608] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [12147.894990] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12147.916008] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12148.054755] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12156.781604] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12168.685507] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12168.689884] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12168.706528] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12168.721264] LustreError: Skipped 1 previous similar message [12171.019807] Lustre: server umount lustre-OST0000 complete [12177.376352] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12177.386604] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12177.403570] Lustre: Skipped 1 previous similar message [12177.415213] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12177.425209] Lustre: Skipped 1 previous similar message [12178.914901] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12178.917527] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12181.471753] Lustre: server umount lustre-MDT0000 complete [12185.707344] LustreError: 586582:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788915309 with bad export cookie 5610517038617038999 [12185.711429] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12185.721962] LustreError: 586582:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12186.108662] Lustre: server umount lustre-MDT0001 complete [12198.413245] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [12201.553735] Key type lgssc unregistered [12201.880207] LNet: 590516:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12201.885942] LNetError: 590516:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12202.920340] LNet: Removed LNI 192.168.204.126@tcp [12203.898893] Key type .llcrypt unregistered [12203.901234] Key type ._llcrypt unregistered [12224.798639] Key type ._llcrypt registered [12224.811369] Key type .llcrypt registered [12224.967283] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [12226.081304] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12226.164321] alg: No test for adler32 (adler32-zlib) [12227.360776] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [12227.595241] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [12229.280067] Key type lgssc registered [12230.854601] Lustre: Echo OBD driver; http://www.lustre.org/ [12240.644251] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12240.667929] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12242.218881] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12248.396922] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [12252.641861] LustreError: 591787:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12252.677466] LustreError: 591787:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12257.769498] LustreError: 591788:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12259.683846] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12260.307582] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12266.076183] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [12277.113181] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12285.446457] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12295.242393] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12295.678811] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12303.325958] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [12312.119873] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12324.906902] loop0: detected capacity change from 0 to 204800000 [12325.206895] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12325.228016] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12325.250779] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12332.565428] loop0: detected capacity change from 0 to 204800000 [12332.810600] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12332.824676] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12332.834510] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12338.388061] loop0: detected capacity change from 0 to 204800000 [12344.558459] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12344.776516] Lustre: fs15246-OST0000: new disk, initializing [12344.785963] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [12344.794916] Lustre: 595383:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12344.847041] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12351.672171] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [12362.794884] loop1: detected capacity change from 0 to 204800000 [12369.969349] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12370.226337] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [12370.315729] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [12370.478978] Lustre: fs15246-MDT0000: new disk, initializing [12370.808205] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12370.841455] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12370.989727] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [12372.066817] Lustre: 596505:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@000000002872df31 x1875813703142784/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788915511 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [12377.088231] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [12381.189752] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [12381.210078] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [12381.275702] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [12408.303756] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12408.317046] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [12409.931881] Lustre: server umount fs15246-MDT0000 complete [12413.442696] LustreError: 593629:0:(ldlm_lib.c:1192:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12418.542796] LustreError: 591783:0:(ldlm_lib.c:1192:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12421.723740] Lustre: server umount fs15246-OST0000 complete [12435.954453] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12435.960841] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12436.013844] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12439.014514] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12439.019364] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12439.026201] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12439.055854] Lustre: Skipped 1 previous similar message [12441.650104] Lustre: server umount lustre-OST0000 complete [12449.250887] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12449.252644] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12449.268819] Lustre: Skipped 1 previous similar message [12449.284555] Lustre: Skipped 1 previous similar message [12452.681461] Lustre: server umount lustre-MDT0000 complete [12454.370886] LustreError: 591788:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12454.391425] LustreError: 591788:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12457.415934] LustreError: 598650:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788915581 with bad export cookie 3395947807335993182 [12457.417910] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12457.452088] LustreError: 598650:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12457.934221] Lustre: server umount lustre-MDT0001 complete [12469.930585] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [12472.840151] Key type lgssc unregistered [12473.188814] LNet: 599604:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12473.202465] LNetError: 599604:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12473.227902] LNet: Removed LNI 192.168.204.126@tcp [12474.340238] Key type .llcrypt unregistered [12474.342753] Key type ._llcrypt unregistered [12493.533568] Key type ._llcrypt registered [12493.544674] Key type .llcrypt registered [12493.688313] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [12495.292176] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12495.324509] alg: No test for adler32 (adler32-zlib) [12496.536428] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [12496.931644] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [12498.687297] Key type lgssc registered [12500.062656] Lustre: Echo OBD driver; http://www.lustre.org/ [12547.662171] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [12561.185891] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [12610.293813] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12610.358969] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12611.813241] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12611.850635] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12611.922992] Lustre: lustre-MDT0000: new disk, initializing [12612.044055] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12612.057454] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12618.450766] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [12633.426113] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12633.600687] Lustre: 604085:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12633.669156] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12633.672533] Lustre: Skipped 1 previous similar message [12633.831101] Lustre: lustre-MDT0001: new disk, initializing [12633.988944] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12634.047480] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12634.066668] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12640.043088] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [12650.734320] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12657.502124] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12666.029047] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12666.366786] Lustre: lustre-OST0000: new disk, initializing [12666.371967] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12666.380298] Lustre: 606057:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12666.493257] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12674.094377] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [12674.783240] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12674.814507] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12675.100133] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12689.026499] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12696.217632] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12696.406128] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12703.254113] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12703.635571] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12710.896053] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12710.900383] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12710.905611] Lustre: Skipped 1 previous similar message [12710.906247] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12712.737173] Lustre: server umount lustre-OST0000 complete [12721.126475] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12721.129700] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12721.139263] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12721.142538] LustreError: Skipped 1 previous similar message [12721.150630] Lustre: Skipped 2 previous similar messages [12723.821413] Lustre: server umount lustre-MDT0000 complete [12728.322792] LustreError: 604078:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788915852 with bad export cookie 14116045599855732240 [12728.336459] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12728.339711] LustreError: 604078:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12728.923856] Lustre: server umount lustre-MDT0001 complete [12764.953281] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [12780.966871] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [12824.077859] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 12539 sec ========= 21:05:44 (1788915944) [12827.051460] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 21:05:47 (1788915947) === [12832.856506] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 21:05:53 (1788915953) === [12888.397568] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [12891.696830] Key type lgssc unregistered [12891.997074] LNet: 612732:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12892.002773] LNetError: 612732:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12892.023944] LNet: Removed LNI 192.168.204.126@tcp [12893.176988] Key type .llcrypt unregistered [12893.178948] Key type ._llcrypt unregistered