[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 592426816 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2480MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524592K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.002386] x2apic enabled [ 0.003008] Switched APIC routing to physical x2apic. [ 0.004016] kvm-guest: setup PV IPIs [ 0.007000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007018] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008013] pid_max: default: 32768 minimum: 301 [ 0.009126] LSM: Security Framework initializing [ 0.010050] Yama: becoming mindful. [ 0.012018] SELinux: Initializing. [ 0.013057] *** VALIDATE selinux *** [ 0.021720] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026605] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027137] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028105] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029118] *** VALIDATE tmpfs *** [ 0.030484] *** VALIDATE proc *** [ 0.031241] *** VALIDATE cgroup *** [ 0.032011] *** VALIDATE cgroup2 *** [ 0.033260] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035102] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037030] Spectre V2 : User space: Vulnerable [ 0.038008] Speculative Store Bypass: Vulnerable [ 0.040952] debug: unmapping init [mem 0xffffffff90259000-0xffffffff90260fff] [ 0.042752] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.043688] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.044024] ... version: 2 [ 0.045012] ... bit width: 48 [ 0.046013] ... generic registers: 4 [ 0.047013] ... value mask: 0000ffffffffffff [ 0.048014] ... max period: 00007fffffffffff [ 0.049024] ... fixed-purpose events: 3 [ 0.050012] ... event mask: 000000070000000f [ 0.051289] rcu: Hierarchical SRCU implementation. [ 0.053362] smp: Bringing up secondary CPUs ... [ 0.054734] x86: Booting SMP configuration: [ 0.055025] .... node #0, CPUs: #1 #2 #3 [ 0.059033] smp: Brought up 1 node, 4 CPUs [ 0.061013] smpboot: Max logical packages: 1 [ 0.062015] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.195102] node 0 deferred pages initialised in 129ms [ 0.200114] devtmpfs: initialized [ 0.202277] x86/mm: Memory block size: 128MB [ 0.208742] gcov: version magic: 0x41383552 [ 0.213361] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.214078] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.215298] pinctrl core: initialized pinctrl subsystem [ 0.217202] [ 0.218009] ************************************************************* [ 0.223015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.228014] ** ** [ 0.233015] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.236014] ** ** [ 0.240015] ** This means that this kernel is built to expose internal ** [ 0.244030] ** IOMMU data structures, which may compromise security on ** [ 0.247022] ** your system. ** [ 0.251013] ** ** [ 0.254015] ** If you see this message and you are not debugging the ** [ 0.257028] ** kernel, report this immediately to your vendor! ** [ 0.260032] ** ** [ 0.264013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.267014] ************************************************************* [ 0.271880] NET: Registered protocol family 16 [ 0.275513] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.278075] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.282086] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.290129] cpuidle: using governor menu [ 0.291908] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.294453] PCI: Using configuration type 1 for base access [ 0.297124] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.309285] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.312056] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.316469] cryptd: max_cpu_qlen set to 1000 [ 0.321164] ACPI: Added _OSI(Module Device) [ 0.323015] ACPI: Added _OSI(Processor Device) [ 0.325019] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.328018] ACPI: Added _OSI(Processor Aggregator Device) [ 0.334315] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.343558] ACPI: Interpreter enabled [ 0.346067] ACPI: PM: (supports S0 S3 S4 S5) [ 0.349014] ACPI: Using IOAPIC for interrupt routing [ 0.351106] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.358397] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.370445] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.374060] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.379027] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.383110] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.392561] acpiphp: Slot [2] registered [ 0.395200] acpiphp: Slot [5] registered [ 0.396187] acpiphp: Slot [6] registered [ 0.398134] acpiphp: Slot [7] registered [ 0.399110] acpiphp: Slot [8] registered [ 0.400431] acpiphp: Slot [9] registered [ 0.402200] acpiphp: Slot [10] registered [ 0.404209] acpiphp: Slot [3] registered [ 0.407141] acpiphp: Slot [4] registered [ 0.409120] acpiphp: Slot [11] registered [ 0.411153] acpiphp: Slot [12] registered [ 0.413143] acpiphp: Slot [13] registered [ 0.415161] acpiphp: Slot [14] registered [ 0.416120] acpiphp: Slot [15] registered [ 0.418119] acpiphp: Slot [16] registered [ 0.419119] acpiphp: Slot [17] registered [ 0.421190] acpiphp: Slot [18] registered [ 0.423165] acpiphp: Slot [19] registered [ 0.424135] acpiphp: Slot [20] registered [ 0.425093] acpiphp: Slot [21] registered [ 0.427122] acpiphp: Slot [22] registered [ 0.428109] acpiphp: Slot [23] registered [ 0.430164] acpiphp: Slot [24] registered [ 0.432155] acpiphp: Slot [25] registered [ 0.435125] acpiphp: Slot [26] registered [ 0.438266] acpiphp: Slot [27] registered [ 0.439000] acpiphp: Slot [28] registered [ 0.441579] acpiphp: Slot [29] registered [ 0.443146] acpiphp: Slot [30] registered [ 0.445164] acpiphp: Slot [31] registered [ 0.447073] PCI host bridge to bus 0000:00 [ 0.448029] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.452032] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.455033] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.459047] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.464039] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.467031] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.469351] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.472514] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.477645] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.490013] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.495078] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.498025] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.502082] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.504025] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.507504] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.511280] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.514049] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.516961] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.524024] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.540035] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.546889] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.552769] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.563000] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.570018] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.607017] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.619930] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.632024] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.644020] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.668022] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.686298] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.697033] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.708023] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.731017] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.752028] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.770015] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.785014] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.806015] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.817100] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.825015] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.834016] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.854014] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.866502] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.874014] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.883014] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.900016] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.919492] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.920379] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.923425] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.925410] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.928219] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.934036] iommu: Default domain type: Passthrough [ 0.935458] SCSI subsystem initialized [ 0.939019] ACPI: bus type USB registered [ 0.942177] usbcore: registered new interface driver usbfs [ 0.945117] usbcore: registered new interface driver hub [ 0.948113] usbcore: registered new device driver usb [ 0.951325] pps_core: LinuxPPS API ver. 1 registered [ 0.954010] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.958062] PTP clock support registered [ 0.961036] EDAC MC: Ver: 3.0.0 [ 0.963149] PCI: Using ACPI for IRQ routing [ 0.966230] NetLabel: Initializing [ 0.968011] NetLabel: domain hash size = 128 [ 0.969010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.972088] NetLabel: unlabeled traffic allowed by default [ 0.975961] vgaarb: loaded [ 0.977313] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.979014] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.985641] clocksource: Switched to clocksource kvm-clock [ 1.099954] VFS: Disk quotas dquot_6.6.0 [ 1.102184] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.104979] *** VALIDATE ramfs *** [ 1.106398] *** VALIDATE hugetlbfs *** [ 1.108254] pnp: PnP ACPI init [ 1.110830] pnp: PnP ACPI: found 6 devices [ 1.128896] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.134621] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.137489] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.140868] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.143443] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.145903] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.149617] NET: Registered protocol family 2 [ 1.153333] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.158920] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.162706] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.168230] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.171536] TCP: Hash tables configured (established 65536 bind 65536) [ 1.174343] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.177184] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.179918] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.182712] NET: Registered protocol family 1 [ 1.188441] RPC: Registered named UNIX socket transport module. [ 1.190941] RPC: Registered udp transport module. [ 1.193186] RPC: Registered tcp transport module. [ 1.195387] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.198188] NET: Registered protocol family 44 [ 1.200371] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.202547] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.204972] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.207880] PCI: CLS 0 bytes, default 64 [ 1.209759] Unpacking initramfs... [ 2.876595] debug: unmapping init [mem 0xffff9a93bcc54000-0xffff9a93bffbffff] [ 2.886895] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.889542] software IO TLB: mapped [mem 0x00000000b8c54000-0x00000000bcc54000] (64MB) [ 2.893900] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.402405] Initialise system trusted keyrings [ 3.404621] Key type blacklist registered [ 3.407036] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.421619] zbud: loaded [ 3.425924] *** VALIDATE nfs *** [ 3.427332] *** VALIDATE nfs4 *** [ 3.429419] pstore: using deflate compression [ 3.433169] Platform Keyring initialized [ 3.561735] NET: Registered protocol family 38 [ 3.563822] Key type asymmetric registered [ 3.565918] Asymmetric key parser 'x509' registered [ 3.568304] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.571952] io scheduler mq-deadline registered [ 3.574521] io scheduler kyber registered [ 3.578074] io scheduler bfq registered [ 3.581366] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.585643] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.591116] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.594339] ACPI: Power Button [PWRF] [ 3.600448] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.609681] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.659216] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.668506] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.694977] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.725211] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.757418] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.765178] Non-volatile memory driver v1.3 [ 3.770342] Linux agpgart interface v0.103 [ 3.812317] virtio_blk virtio1: [vda] 145896 512-byte logical blocks (74.7 MB/71.2 MiB) [ 3.817969] vda: detected capacity change from 0 to 74698752 [ 3.842000] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.848819] vdb: detected capacity change from 0 to 1073741824 [ 3.870546] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.874304] vdc: detected capacity change from 0 to 2621440000 [ 3.894462] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.898133] vdd: detected capacity change from 0 to 2621440000 [ 3.915203] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.918354] vde: detected capacity change from 0 to 4294967296 [ 3.938891] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.942203] vdf: detected capacity change from 0 to 4294967296 [ 3.952409] libphy: Fixed MDIO Bus: probed [ 3.960989] usbcore: registered new interface driver usbserial_generic [ 3.963379] usbserial: USB Serial support registered for generic [ 3.965098] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.970284] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.971795] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.973976] mousedev: PS/2 mouse device common for all mice [ 3.978430] rtc_cmos 00:05: RTC can wake from S4 [ 3.978966] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.982061] rtc_cmos 00:05: registered as rtc0 [ 3.989010] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.988412] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.000168] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.004200] intel_pstate: CPU model not supported [ 4.012735] hid: raw HID events driver (C) Jiri Kosina [ 4.015611] usbcore: registered new interface driver usbhid [ 4.018529] usbhid: USB HID core driver [ 4.020534] drop_monitor: Initializing network drop monitor service [ 4.023803] Initializing XFRM netlink socket [ 4.026507] NET: Registered protocol family 10 [ 4.030369] Segment Routing with IPv6 [ 4.032461] NET: Registered protocol family 17 [ 4.035488] mpls_gso: MPLS GSO support [ 4.042983] RAS: Correctable Errors collector initialized. [ 4.045399] AVX version of gcm_enc/dec engaged. [ 4.047241] AES CTR mode by8 optimization enabled [ 4.152182] sched_clock: Marking stable (4152156269, 0)->(5112545880, -960389611) [ 4.158973] registered taskstats version 1 [ 4.162820] Loading compiled-in X.509 certificates [ 4.169637] zswap: loaded using pool lzo/zbud [ 4.216093] Key type big_key registered [ 4.237920] Key type encrypted registered [ 4.239657] ima: No TPM chip found, activating TPM-bypass! [ 4.242431] ima: Allocated hash algorithm: sha1 [ 4.245138] ima: No architecture policies found [ 4.247770] evm: Initialising EVM extended attributes: [ 4.249599] evm: security.selinux [ 4.250609] evm: security.ima [ 4.252104] evm: security.capability [ 4.253440] evm: HMAC attrs: 0x1 [ 4.256203] rtc_cmos 00:05: setting system clock to 2026-08-14 21:42:23 UTC (1786743743) [ 4.263729] debug: unmapping init [mem 0xffffffff91203000-0xffffffff913fffff] [ 4.267465] debug: unmapping init [mem 0xffffffff8ff82000-0xffffffff90258fff] [ 4.276084] Write protecting the kernel read-only data: 28672k [ 4.280663] debug: unmapping init [mem 0xffffffff8e603000-0xffffffff8e7fffff] [ 4.283801] debug: unmapping init [mem 0xffffffff8ef14000-0xffffffff8effffff] [ 4.327499] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.336244] systemd[1]: Detected virtualization kvm. [ 4.338241] systemd[1]: Detected architecture x86-64. [ 4.340534] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.374487] systemd[1]: No hostname configured. [ 4.376761] systemd[1]: Set hostname to . [ 4.379351] random: systemd: uninitialized urandom read (16 bytes read) [ 4.382583] systemd[1]: Initializing machine ID from random generator. [ 4.450942] random: ln: uninitialized urandom read (6 bytes read) [ 4.568890] random: systemd: uninitialized urandom read (16 bytes read) [ 4.571955] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 4.582924] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 4.592471] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Local File Systems. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... Starting Create Volatile Files and Directories... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Local Encrypted Volumes. Starting Journal Service... [ OK ] Reached target Paths. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Slices. [ OK ] Reached target Timers. Starting Apply Kernel Variables... Starting Setup Virtual Console... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 5.435510] device-mapper: uevent: version 1.0.3 [ 5.439056] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [[ 6.111436] random: fast init done  OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 6.316620] virtio_net virtio0 ens2: renamed from eth0 [ 6.442854] scsi host0: ata_piix [ 6.548211] scsi host1: ata_piix [ 6.593882] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 6.598574] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 11.497861] dracut-initqueue[587]: RTNETLINK answers: File exists [ 11.760646] random: crng init done [ 11.761859] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 12.488679] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 14.049956] printk: systemd: 25 output lines suppressed due to ratelimiting [ 14.349933] SELinux: Disabled at runtime. [ 14.410590] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 14.418605] systemd[1]: Detected virtualization kvm. [ 14.420646] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 15.079634] systemd[1]: initrd-switch-root.service: Succeeded. [ 15.082424] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 15.088268] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 15.092667] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 15.097299] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 15.110110] systemd[1]: Starting Journal Service... Starting Journal Service... [ 15.115973] systemd[1]: Listening on initctl Compatibility Named Pipe. [ OK ] Listening on initctl Compatibility Named Pipe. Starting Remount Root and Kernel File Systems... [ OK ] Stopped target Switch Root. [ OK ] Listening on udev Control Socket. [ OK ] Created slice User and Session Slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on RPCbind Server Activation Socket. Mounting Huge Pages File System... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Reached target rpc_pipefs.target. Starting udev Coldplug all Devices... [ OK ] Created slice system-getty.slice. Mounting POSIX Message Queue File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target Slices. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target RPC Port Mapper. Starting Create list of required[ 15.351177] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS st…ce nodes for the current kernel... Starting Apply Kernel Variables... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on Process Core Dump Socket. Mounting Kernel Debug File System... [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 15.860360] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 16.317618] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 16.378853] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 16.561458] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 16.578824] EDAC sbridge: Ver: 1.1.2 [ 18.466134] Key type dns_resolver registered [ 18.790463] NFS: Registering the id_resolver key type [ 18.792787] Key type id_resolver registered [ 18.794362] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started irqbalance daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Authorization Manager. [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg651-server login: [ 61.675516] libcfs: loading out-of-tree module taints kernel. [ 61.769048] Key type ._llcrypt registered [ 61.770746] Key type .llcrypt registered [ 61.920450] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_hostid [ 83.309743] hrtimer: interrupt took 6734118 ns [ 86.679593] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 89.298359] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 89.330537] alg: No test for adler32 (adler32-zlib) [ 91.343517] Lustre: Lustre: Build Version: 2.17.57_1_g59aa375 [ 92.764417] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 94.695174] Key type lgssc registered [ 96.882367] Lustre: Echo OBD driver; http://www.lustre.org/ [ 114.113311] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 154.201944] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 166.104830] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 166.180251] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 167.464498] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 167.504489] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 167.581967] Lustre: lustre-MDT0000: new disk, initializing [ 167.700356] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 167.717977] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 172.569218] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 185.893027] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 186.006831] Lustre: 6512:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 186.044873] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 186.049091] Lustre: Skipped 1 previous similar message [ 186.131070] Lustre: lustre-MDT0001: new disk, initializing [ 186.212815] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 186.234872] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 186.244626] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 190.325866] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 195.228792] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 204.433828] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 204.779393] Lustre: lustre-OST0000: new disk, initializing [ 204.783831] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 204.788849] Lustre: 8450:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 204.877384] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 212.039391] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 212.058647] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 212.100724] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 212.202418] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 227.731351] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 227.857709] Lustre: lustre-OST0001: new disk, initializing [ 227.863262] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 227.867662] Lustre: 9523:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 227.952484] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 233.859452] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 236.096844] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 236.107218] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 236.171627] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 245.430983] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 251.968518] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 258.177778] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing check_logdir /tmp/testlogs/ [ 262.866962] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing yml_node [ 266.586741] Lustre: DEBUG MARKER: Client: 2.17.57.1 [ 269.928386] Lustre: DEBUG MARKER: MDS: 2.17.57.1 [ 273.247947] Lustre: DEBUG MARKER: OSS: 2.17.57.1 [ 275.419335] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Aug 14 17:46:52 EDT 2026 [ 294.596647] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 296.364505] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 312.802441] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 312.812474] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 312.823662] Lustre: Skipped 2 previous similar messages [ 317.929635] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 317.939920] Lustre: Skipped 6 previous similar messages [ 318.849810] Lustre: server umount lustre-MDT0000 complete [ 323.230109] LustreError: 10339:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786744062 with bad export cookie 16820912563094732299 [ 323.240853] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 323.247139] LustreError: 10339:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 328.161408] LustreError: 8075:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 328.161702] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 328.174043] LustreError: 8075:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 328.194962] Lustre: Skipped 1 previous similar message [ 328.211090] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 328.220100] Lustre: Skipped 1 previous similar message [ 329.685423] Lustre: server umount lustre-MDT0001 complete [ 334.250995] Lustre: server umount lustre-OST0000 complete [ 338.367922] Lustre: server umount lustre-OST0001 complete [ 344.092247] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_hostid [ 353.244343] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 387.504144] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 387.759593] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 387.794703] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 387.936784] Lustre: lustre-MDT0000: new disk, initializing [ 388.032853] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 388.056953] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 392.359406] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 403.946636] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 404.104849] Lustre: 15079:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 404.156232] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 404.160865] Lustre: Skipped 1 previous similar message [ 404.334434] Lustre: lustre-MDT0001: new disk, initializing [ 404.469642] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 404.514037] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 404.527662] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 409.508419] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 419.608191] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 425.307042] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 433.572099] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 434.064153] Lustre: lustre-OST0000: new disk, initializing [ 434.068439] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 434.074433] Lustre: 17045:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 434.210937] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 435.284050] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 435.306941] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 435.507956] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 441.223895] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 452.791145] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 458.091097] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 458.285383] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 462.572231] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 462.823554] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 466.408096] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 466.426610] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 466.441561] Lustre: Skipped 2 previous similar messages [ 466.448929] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 466.458355] Lustre: Skipped 1 previous similar message [ 471.526128] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 471.535509] Lustre: Skipped 1 previous similar message [ 479.199172] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 479.459514] Lustre: server umount lustre-OST0000 complete [ 486.368551] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 486.384354] LustreError: Skipped 1 previous similar message [ 486.388277] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 486.414649] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 486.422476] Lustre: Skipped 2 previous similar messages [ 489.507416] Lustre: server umount lustre-MDT0000 complete [ 492.006094] LustreError: 15934:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 492.016616] LustreError: 15934:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 493.315803] LustreError: 15069:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786744232 with bad export cookie 16820912563094733839 [ 493.329036] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 493.334269] LustreError: 15069:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 493.788651] Lustre: server umount lustre-MDT0001 complete [ 498.226366] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 17:50:35 (1786744235) [ 500.823842] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 502.767742] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 17:50:40 (1786744240) [ 511.052463] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 511.766086] LustreError: 19457:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 511.845433] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 516.929337] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 517.090509] LustreError: 19458:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 522.216653] LustreError: 19457:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 526.182443] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 531.784728] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 539.795206] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 545.695540] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 553.058118] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 560.063397] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 568.432622] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 581.094872] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 581.100398] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 581.119494] Lustre: Skipped 1 previous similar message [ 581.130778] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 581.140434] Lustre: Skipped 2 previous similar messages [ 585.828905] Lustre: server umount lustre-OST0000 complete [ 591.329911] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 591.341955] Lustre: Skipped 1 previous similar message [ 604.130831] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 604.425633] Lustre: server umount lustre-MDT0000 complete [ 606.688746] LustreError: 19458:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 606.719559] LustreError: 19458:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 608.653648] LustreError: 19437:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786744347 with bad export cookie 16820912563094734721 [ 608.669391] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 609.240302] Lustre: server umount lustre-MDT0001 complete [ 623.186446] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 626.258801] Key type lgssc unregistered [ 626.564596] LNet: 22995:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 626.575410] LNetError: 22995:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 627.622737] LNet: Removed LNI 192.168.206.151@tcp [ 628.738235] Key type .llcrypt unregistered [ 628.742489] Key type ._llcrypt unregistered [ 653.492867] Key type ._llcrypt registered [ 653.495109] Key type .llcrypt registered [ 653.610937] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 654.784862] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 654.897680] alg: No test for adler32 (adler32-zlib) [ 656.020460] Lustre: Lustre: Build Version: 2.17.57_1_g59aa375 [ 656.339404] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 658.007253] Key type lgssc registered [ 659.260433] Lustre: Echo OBD driver; http://www.lustre.org/ [ 669.138161] loop: module loaded [ 669.187152] loop0: detected capacity change from 0 to 204800000 [ 669.441275] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 669.469463] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 669.952417] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 677.287621] loop0: detected capacity change from 0 to 204800000 [ 677.453967] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 677.463565] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 677.471866] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 683.075881] loop0: detected capacity change from 0 to 204800000 [ 689.277092] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 689.310081] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 690.514562] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 690.556819] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 690.657438] Lustre: lustre-MDT0000: new disk, initializing [ 690.741603] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 690.758995] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 696.493593] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 706.616284] loop1: detected capacity change from 0 to 204800000 [ 713.603380] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 713.966366] Lustre: lustre-OST07c6: new disk, initializing [ 713.970644] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 713.974902] Lustre: Skipped 1 previous similar message [ 713.982890] Lustre: 26183:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 714.088464] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 720.425841] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 723.057918] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 723.075978] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 723.132969] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 726.266161] Lustre: 25170:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 726.277324] Lustre: 25170:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 726.287752] Lustre: 25170:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 726.294916] Lustre: 25170:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 726.308928] Lustre: 25170:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 726.322696] Lustre: 25170:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 744.074768] Lustre: 25170:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 744.082594] Lustre: 25170:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 744.088601] Lustre: 25170:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 744.103832] Lustre: 25170:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 744.115456] Lustre: 25170:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 744.125775] Lustre: 25170:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 750.002865] Lustre: 25171:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 750.013725] Lustre: 25171:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 750.019437] Lustre: 25171:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 750.026031] Lustre: 25171:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 750.033509] Lustre: 25171:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 750.039653] Lustre: 25171:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 750.046249] Lustre: 25171:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 750.052545] Lustre: 25171:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 750.057587] Lustre: 25171:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 750.063353] Lustre: 25171:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 750.068522] Lustre: 25171:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 750.074759] Lustre: 25171:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 753.251033] Lustre: 25170:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 753.261404] Lustre: 25170:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 753.264807] Lustre: 25170:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 753.268659] Lustre: 25170:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 753.272279] Lustre: 25170:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 753.276838] Lustre: 25170:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 753.280600] Lustre: 25170:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 753.284971] Lustre: 25170:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 753.290597] Lustre: 25170:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 753.298415] Lustre: 25170:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 753.306631] Lustre: 25170:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 753.310828] Lustre: 25170:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 758.071821] Lustre: 26764:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 758.085503] Lustre: 26764:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 758.094524] Lustre: 26764:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 758.103454] Lustre: 26764:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 758.109958] Lustre: 26764:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 758.118485] Lustre: 26764:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 758.125566] Lustre: 26764:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 758.134048] Lustre: 26764:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 758.145325] Lustre: 26764:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 758.151856] Lustre: 26764:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 758.165580] Lustre: 26764:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 758.174905] Lustre: 26764:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 769.478348] Lustre: 26764:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 769.493384] Lustre: 26764:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 5 previous similar messages [ 769.500322] Lustre: 26764:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 769.509748] Lustre: 26764:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 769.524056] Lustre: 26764:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 769.529104] Lustre: 26764:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 769.536487] Lustre: 26764:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 769.547027] Lustre: 26764:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 769.557220] Lustre: 26764:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 769.563307] Lustre: 26764:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 769.570431] Lustre: 26764:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 769.582380] Lustre: 26764:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 820.198469] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 820.202326] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 820.224981] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 823.013470] Lustre: server umount lustre-OST07c6 complete [ 833.952746] Lustre: server umount lustre-MDT0000 complete [ 852.540711] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 855.583347] Key type lgssc unregistered [ 855.848840] LNet: 28742:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 855.857809] LNetError: 28742:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 855.880689] LNet: Removed LNI 192.168.206.151@tcp [ 856.728688] Key type .llcrypt unregistered [ 856.731768] Key type ._llcrypt unregistered [ 878.103820] Key type ._llcrypt registered [ 878.107290] Key type .llcrypt registered [ 878.241591] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 879.480249] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 879.582075] alg: No test for adler32 (adler32-zlib) [ 880.619691] Lustre: Lustre: Build Version: 2.17.57_1_g59aa375 [ 880.895928] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 882.600519] Key type lgssc registered [ 883.760180] Lustre: Echo OBD driver; http://www.lustre.org/ [ 891.713309] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 891.741332] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 893.096177] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 897.406443] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 903.651892] LustreError: 30015:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 903.678503] LustreError: 30015:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 905.828445] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 906.282155] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 910.929706] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 918.737747] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 924.971290] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 932.256733] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 932.548559] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 938.743542] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 938.969527] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 946.314276] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 964.584938] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 964.599383] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 964.618314] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 966.624310] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 966.633736] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 966.636790] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 969.945736] Lustre: server umount lustre-OST0000 complete [ 976.869335] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 976.873682] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 976.878653] Lustre: Skipped 1 previous similar message [ 976.890324] Lustre: Skipped 2 previous similar messages [ 980.416449] Lustre: server umount lustre-MDT0000 complete [ 981.988143] LustreError: 31019:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 982.039264] LustreError: 31019:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 984.134314] LustreError: 29996:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786744723 with bad export cookie 16811461142378028633 [ 984.143376] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 984.143942] LustreError: 29996:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 984.502649] Lustre: server umount lustre-MDT0001 complete [ 995.909515] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 998.989877] Key type lgssc unregistered [ 999.347256] LNet: 33754:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 999.363425] LNetError: 33754:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 999.388048] LNet: Removed LNI 192.168.206.151@tcp [ 1000.346185] Key type .llcrypt unregistered [ 1000.348311] Key type ._llcrypt unregistered [ 1019.054233] Key type ._llcrypt registered [ 1019.061133] Key type .llcrypt registered [ 1019.161795] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 1020.040662] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1020.100323] alg: No test for adler32 (adler32-zlib) [ 1021.167524] Lustre: Lustre: Build Version: 2.17.57_1_g59aa375 [ 1021.425884] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 1023.095764] Key type lgssc registered [ 1024.089457] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1031.407202] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1031.446033] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1032.949517] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1037.654535] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 1043.433663] LustreError: 35040:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1043.447454] LustreError: 35040:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1046.447100] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1046.798905] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1051.230936] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 1059.003663] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1065.428298] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1072.425747] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1072.847991] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1078.001399] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1079.715270] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 1087.365188] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1101.794885] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1101.802826] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1101.808670] Lustre: Skipped 1 previous similar message [ 1101.820490] Lustre: Skipped 1 previous similar message [ 1106.921520] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1106.931538] Lustre: Skipped 1 previous similar message [ 1108.023493] Lustre: server umount lustre-OST0000 complete [ 1112.034717] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1112.047264] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1112.057877] Lustre: Skipped 1 previous similar message [ 1117.154119] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1117.165798] Lustre: Skipped 1 previous similar message [ 1122.292570] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1122.305129] Lustre: Skipped 1 previous similar message [ 1126.369227] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1126.635206] Lustre: server umount lustre-MDT0000 complete [ 1127.392346] LustreError: 35040:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1127.426506] LustreError: 35040:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1130.803250] LustreError: 37541:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786744870 with bad export cookie 925522767392344788 [ 1130.811416] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1130.815158] LustreError: 37541:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1131.616209] Lustre: server umount lustre-MDT0001 complete [ 1144.371349] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 1147.313276] Key type lgssc unregistered [ 1147.611209] LNet: 38613:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1147.621773] LNetError: 38613:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1147.640331] LNet: Removed LNI 192.168.206.151@tcp [ 1148.515874] Key type .llcrypt unregistered [ 1148.522141] Key type ._llcrypt unregistered [ 1177.734848] Key type ._llcrypt registered [ 1177.738923] Key type .llcrypt registered [ 1177.968189] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 1179.312636] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1179.397315] alg: No test for adler32 (adler32-zlib) [ 1180.751765] Lustre: Lustre: Build Version: 2.17.57_1_g59aa375 [ 1181.017411] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 1182.735866] Key type lgssc registered [ 1183.874583] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1192.278829] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1192.303627] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1193.816806] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1198.262673] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 1204.197899] LustreError: 40046:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1204.217165] LustreError: 40046:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1206.927435] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1207.301167] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1211.353296] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 1219.551601] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1225.894955] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1233.799490] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1234.234973] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1239.354825] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1240.251739] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 1247.970624] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1257.953695] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1257.962584] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1257.966693] Lustre: Skipped 1 previous similar message [ 1259.951409] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.51@tcp (stopping) [ 1263.078384] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1263.100792] Lustre: Skipped 2 previous similar messages [ 1263.399432] Lustre: server umount lustre-MDT0000 complete [ 1264.998391] LustreError: 41324:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1268.064911] LustreError: 40027:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786745007 with bad export cookie 4749678208881692300 [ 1268.080408] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1268.084367] LustreError: 40027:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1268.192759] LustreError: 40040:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1268.193125] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1268.252230] Lustre: Skipped 2 previous similar messages [ 1268.270285] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1273.314096] LustreError: 40042:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1273.315614] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1273.344633] LustreError: 40042:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1273.352151] Lustre: Skipped 1 previous similar message [ 1274.457294] Lustre: server umount lustre-MDT0001 complete [ 1289.958435] Lustre: server umount lustre-OST0000 complete [ 1305.531568] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 1308.440618] Key type lgssc unregistered [ 1308.772993] LNet: 43905:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1308.786377] LNetError: 43905:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1308.802826] LNet: Removed LNI 192.168.206.151@tcp [ 1309.560683] Key type .llcrypt unregistered [ 1309.563773] Key type ._llcrypt unregistered [ 1334.223900] Key type ._llcrypt registered [ 1334.227281] Key type .llcrypt registered [ 1334.323398] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 1335.523569] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1335.634956] alg: No test for adler32 (adler32-zlib) [ 1336.802208] Lustre: Lustre: Build Version: 2.17.57_1_g59aa375 [ 1337.062949] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 1338.751189] Key type lgssc registered [ 1339.729893] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1346.934690] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1346.955628] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1348.359419] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1353.179476] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 1357.797096] LustreError: 45322:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1357.847298] LustreError: 45322:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1361.910362] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1362.299311] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1367.352772] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 1374.438314] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1380.837816] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1387.581047] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1388.108323] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1393.270886] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1394.459057] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 1402.185471] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1413.090848] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1413.092182] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1413.103372] Lustre: Skipped 1 previous similar message [ 1413.117194] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1413.283793] Lustre: server umount lustre-OST0000 complete [ 1433.315056] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1433.338517] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1433.348932] Lustre: Skipped 2 previous similar messages [ 1437.886714] Lustre: server umount lustre-MDT0000 complete [ 1438.694319] LustreError: 46052:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1438.728419] LustreError: 46052:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1442.398632] LustreError: 45304:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786745181 with bad export cookie 963640766474817238 [ 1442.419926] LustreError: 45304:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1442.420641] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1448.894640] Lustre: server umount lustre-MDT0001 complete [ 1461.271971] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 1464.089965] Key type lgssc unregistered [ 1464.390976] LNet: 49036:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1464.398277] LNetError: 49036:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1464.456926] LNet: Removed LNI 192.168.206.151@tcp [ 1465.417725] Key type .llcrypt unregistered [ 1465.421046] Key type ._llcrypt unregistered [ 1490.836602] Key type ._llcrypt registered [ 1490.838753] Key type .llcrypt registered [ 1490.926508] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 1492.306917] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1492.355163] alg: No test for adler32 (adler32-zlib) [ 1493.473492] Lustre: Lustre: Build Version: 2.17.57_1_g59aa375 [ 1493.794683] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 1495.503162] Key type lgssc registered [ 1496.606392] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1504.936465] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1504.967707] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1506.413135] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1510.713654] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 1516.006293] LustreError: 50454:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1516.027961] LustreError: 50454:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1518.768188] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1519.164184] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1523.545591] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 1530.700869] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1536.312708] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1543.508312] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1543.959939] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1549.839894] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 1551.222194] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1556.937449] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1562.781449] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1566.223688] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1578.497557] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-08-1418h08m35s [ 1580.072040] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1582.050624] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1582.058181] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1582.061296] Lustre: Skipped 1 previous similar message [ 1582.080322] Lustre: Skipped 2 previous similar messages [ 1583.151232] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1583.155680] Lustre: Skipped 2 previous similar messages [ 1585.035047] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.51@tcp (stopping) [ 1589.216519] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1589.230752] Lustre: Skipped 3 previous similar messages [ 1595.363782] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1595.370687] Lustre: Skipped 3 previous similar messages [ 1596.383573] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1596.588891] Lustre: server umount lustre-MDT0000 complete [ 1597.284440] LustreError: 52904:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1601.509588] LustreError: 52904:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1601.549143] LustreError: 52904:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1601.600566] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1607.661287] LustreError: 50455:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1607.679174] LustreError: 50455:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1609.070468] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1609.152051] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1609.379574] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1609.465903] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1610.425569] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1613.299623] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 1616.866891] Lustre: DEBUG MARKER: Wait for df (36774) ... [ 1618.576490] Lustre: DEBUG MARKER: done [ 1625.058990] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1625.071182] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1625.089219] Lustre: Skipped 1 previous similar message [ 1625.102990] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1626.086875] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1629.610193] Lustre: server umount lustre-OST0000 complete [ 1636.320338] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1636.333549] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1636.385082] Lustre: Skipped 1 previous similar message [ 1640.197706] Lustre: server umount lustre-MDT0000 complete [ 1643.576736] LustreError: 50433:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786745382 with bad export cookie 8516887865015402735 [ 1643.590124] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1643.590448] LustreError: 50433:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1644.013141] Lustre: server umount lustre-MDT0001 complete [ 1655.589365] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 1657.943870] Key type lgssc unregistered [ 1658.223459] LNet: 55383:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1658.228201] LNetError: 55383:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1658.237704] LNet: Removed LNI 192.168.206.151@tcp [ 1658.897117] Key type .llcrypt unregistered [ 1658.899484] Key type ._llcrypt unregistered [ 1697.567753] Key type ._llcrypt registered [ 1697.569358] Key type .llcrypt registered [ 1697.664932] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 1698.820547] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1698.857045] alg: No test for adler32 (adler32-zlib) [ 1699.934403] Lustre: Lustre: Build Version: 2.17.57_1_g59aa375 [ 1700.142727] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 1701.807225] Key type lgssc registered [ 1702.754456] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1711.853462] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1711.887780] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1713.455737] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1713.496228] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1713.774167] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1718.789372] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 1730.415475] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1730.564341] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1730.614952] Lustre: 57594:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1730.904679] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1735.903621] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 1744.518939] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1751.584646] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1759.213654] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1759.387232] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1759.601662] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1765.317822] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 1766.040838] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1773.189332] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1780.758955] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-08-1418h11m57s [ 1782.815146] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1790.818513] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1793.540181] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1793.618936] LustreError: 57599:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1813.910948] Lustre: lustre-MDT0000: Client ad94124f-f4db-492c-98cb-b480dda742de (at 192.168.206.51@tcp) reconnecting [ 1833.623121] LustreError: 57599:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1836.763902] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1839.694466] Lustre: DEBUG MARKER: done [ 1847.778726] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1847.787652] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1847.806030] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1848.295216] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1848.312149] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1848.319378] Lustre: Skipped 1 previous similar message [ 1851.694361] Lustre: server umount lustre-OST0000 complete [ 1858.528697] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1858.534174] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1858.542147] Lustre: Skipped 1 previous similar message [ 1863.440889] Lustre: server umount lustre-MDT0000 complete [ 1866.601819] LustreError: 60356:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786745605 with bad export cookie 6615656405141096568 [ 1866.605535] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1866.611094] LustreError: 60356:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1866.879467] Lustre: server umount lustre-MDT0001 complete [ 1877.107963] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 1879.582625] Key type lgssc unregistered [ 1879.866215] LNet: 61929:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1879.875854] LNetError: 61929:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1879.891926] LNet: Removed LNI 192.168.206.151@tcp [ 1880.546608] Key type .llcrypt unregistered [ 1880.550068] Key type ._llcrypt unregistered [ 1931.159275] Key type ._llcrypt registered [ 1931.161754] Key type .llcrypt registered [ 1931.348212] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 1933.809269] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1933.849903] alg: No test for adler32 (adler32-zlib) [ 1935.158917] Lustre: Lustre: Build Version: 2.17.57_1_g59aa375 [ 1935.382046] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 1937.071181] Key type lgssc registered [ 1939.019908] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1943.680601] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1943.716822] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1944.960458] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1944.986572] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1945.170603] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1946.090522] Lustre: Failing over lustre-MDT0000 [ 1946.398260] Lustre: server umount lustre-MDT0000 complete [ 1954.519367] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 18:14:52 (1786745692) [ 1962.177103] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1962.902292] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1968.017570] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 1977.080316] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1977.240837] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1977.294317] Lustre: 64739:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1977.635754] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1982.396549] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 1991.332452] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1998.254851] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2006.209925] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2006.408620] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2006.665607] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2014.720469] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 2015.917537] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2023.843802] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2030.425960] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2039.268934] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2039.276143] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2039.292938] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2041.831399] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2041.834767] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2041.846406] Lustre: Skipped 1 previous similar message [ 2041.873224] Lustre: Skipped 1 previous similar message [ 2045.008079] Lustre: server umount lustre-MDT0000 complete [ 2049.046379] LustreError: 64733:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786745788 with bad export cookie 6435086152112335250 [ 2049.051250] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2049.063558] LustreError: 64733:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2049.566248] Lustre: server umount lustre-MDT0001 complete [ 2053.381141] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2058.282572] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2067.165855] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2068.447774] Lustre: 63664:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786745791/real 1786745791] req@00000000eb5b1f83 x1873538631902976/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786745807 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2068.485731] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2070.495138] Lustre: 63663:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786745793/real 1786745793] req@00000000a113ddc7 x1873538631903104/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786745809 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2073.570119] Lustre: 63661:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786745796/real 1786745796] req@000000003d9b7f66 x1873538631903232/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786745812 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2074.655745] LustreError: 63660:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000f6bae70c x1873538631904000/t0(0) o250->MGC192.168.206.151@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2075.095913] LustreError: 66587:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2075.124975] LustreError: 66587:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2075.226810] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2075.394920] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2076.258805] Lustre: 63661:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786745798/real 1786745798] req@00000000c3414575 x1873538631903488/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786745814 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2077.228227] LustreError: 66588:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2078.242881] LustreError: 68512:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2078.272217] LustreError: 68512:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2080.516699] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 2081.251845] Lustre: 63663:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786745803/real 1786745803] req@0000000092ed291c x1873538631903616/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786745819 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2083.309962] LustreError: 68511:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2083.337972] LustreError: 68511:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2088.418972] LustreError: 66587:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2088.465165] LustreError: 66587:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2090.181732] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2090.540299] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2095.291875] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 2095.592918] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2097.643785] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2103.450899] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2109.354517] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2116.071922] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2116.082543] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2116.150460] LustreError: 66587:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2116.192591] Lustre: server umount lustre-MDT0000 complete [ 2120.411515] LustreError: 66594:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786745859 with bad export cookie 6435086152112337525 [ 2120.417142] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2120.423466] LustreError: 66594:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2120.804644] Lustre: server umount lustre-MDT0001 complete [ 2128.159282] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2139.615312] Lustre: 63662:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786745862/real 1786745862] req@0000000067f5b392 x1873538631930752/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786745878 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2139.671910] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2139.685141] Lustre: Skipped 2 previous similar messages [ 2139.777592] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2145.169673] LustreError: 66587:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2145.192948] LustreError: 66587:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2145.326838] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2145.400285] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2150.806615] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 2158.605417] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2159.425215] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2164.486458] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 2165.245083] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2172.683295] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2178.682269] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2185.699205] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2185.702383] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2185.713981] Lustre: Skipped 2 previous similar messages [ 2185.739717] Lustre: Skipped 3 previous similar messages [ 2190.825091] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2190.833422] Lustre: Skipped 2 previous similar messages [ 2191.637592] Lustre: server umount lustre-MDT0000 complete [ 2195.384989] LustreError: 64731:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786745934 with bad export cookie 6435086152112338904 [ 2195.390771] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2195.413737] LustreError: 64731:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2195.637047] Lustre: server umount lustre-MDT0001 complete [ 2202.450356] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2206.063798] Lustre: DEBUG MARKER: files compared the same [ 2218.045423] Lustre: server umount lustre-OST0000 complete [ 2235.230856] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 2237.953922] Key type lgssc unregistered [ 2238.232077] LNet: 74471:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2238.241442] LNetError: 74471:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2238.266269] LNet: Removed LNI 192.168.206.151@tcp [ 2239.096865] Key type .llcrypt unregistered [ 2239.103276] Key type ._llcrypt unregistered [ 2269.444100] Key type ._llcrypt registered [ 2269.451136] Key type .llcrypt registered [ 2269.686822] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 2271.082211] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2271.238201] alg: No test for adler32 (adler32-zlib) [ 2272.336956] Lustre: Lustre: Build Version: 2.17.57_1_g59aa375 [ 2272.702262] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 2274.503168] Key type lgssc registered [ 2275.928863] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2285.158327] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2285.189088] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2286.603714] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2291.162817] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug +malloc all [ 2295.777743] LustreError: 75891:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2295.791885] LustreError: 75891:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2299.183472] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2299.426225] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2303.676929] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug +malloc all [ 2311.347186] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2317.319213] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2323.956843] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2324.265493] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2330.308593] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug +malloc all [ 2331.433616] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2337.853657] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2350.054929] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2350.057528] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2350.060253] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2350.084364] Lustre: Skipped 1 previous similar message [ 2351.760508] Lustre: server umount lustre-OST0000 complete [ 2355.680031] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2355.695558] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2355.705739] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2355.709038] Lustre: Skipped 1 previous similar message [ 2360.296659] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2360.296728] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2360.328792] Lustre: Skipped 1 previous similar message [ 2361.837286] Lustre: server umount lustre-MDT0000 complete [ 2365.408586] LustreError: 75885:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2365.437261] LustreError: 75885:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2365.862063] LustreError: 75872:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786746105 with bad export cookie 10196344740508062733 [ 2365.871857] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2366.217558] Lustre: server umount lustre-MDT0001 complete [ 2379.765388] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 2383.368278] Key type lgssc unregistered [ 2383.605735] LNet: 79424:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2383.613808] LNetError: 79424:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2383.629735] LNet: Removed LNI 192.168.206.151@tcp [ 2384.530457] Key type .llcrypt unregistered [ 2384.533556] Key type ._llcrypt unregistered [ 2408.611332] Key type ._llcrypt registered [ 2408.622617] Key type .llcrypt registered [ 2408.735946] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 2409.719320] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2409.750997] alg: No test for adler32 (adler32-zlib) [ 2410.802462] Lustre: Lustre: Build Version: 2.17.57_1_g59aa375 [ 2411.052696] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 2412.703161] Key type lgssc registered [ 2414.281411] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2423.642411] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2423.656394] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2439.263476] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2444.383756] LustreError: 80817:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.151@tcp: failed processing log, type 4: rc = -110 [ 2471.135229] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2477.895423] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 2485.194383] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2493.769848] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2493.936608] Lustre: *** cfs_fail_loc=706, val=0*** [ 2494.345864] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2494.426102] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2498.386545] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 2499.563075] LustreError: 80842:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2499.588686] LustreError: 80842:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2504.674369] LustreError: 80843:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2506.647781] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2506.724231] LustreError: 81901:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2506.749678] LustreError: 81901:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2506.983857] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2511.299567] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 2518.952516] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2524.346215] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2527.715779] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2527.720927] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2527.731193] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2527.751185] Lustre: Skipped 1 previous similar message [ 2532.833574] LustreError: 80834:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2533.082582] Lustre: server umount lustre-OST0000 complete [ 2537.952027] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2537.959053] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2537.960201] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2537.960206] Lustre: Skipped 1 previous similar message [ 2537.991575] Lustre: Skipped 1 previous similar message [ 2543.081045] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2543.090736] Lustre: Skipped 2 previous similar messages [ 2548.193083] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2548.199602] Lustre: Skipped 1 previous similar message [ 2551.264962] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2553.313208] LustreError: 81902:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2553.324101] LustreError: 81902:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2553.413262] Lustre: server umount lustre-MDT0000 complete [ 2556.424811] LustreError: 80823:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786746295 with bad export cookie 1226340190447149722 [ 2556.444883] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2556.714408] Lustre: server umount lustre-MDT0001 complete [ 2568.082484] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 2571.144078] Key type lgssc unregistered [ 2571.460962] LNet: 84434:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2571.470789] LNetError: 84434:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2571.511837] LNet: Removed LNI 192.168.206.151@tcp [ 2572.509208] Key type .llcrypt unregistered [ 2572.513784] Key type ._llcrypt unregistered [ 2596.729172] Key type ._llcrypt registered [ 2596.730614] Key type .llcrypt registered [ 2596.809652] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 2597.910958] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2597.977041] alg: No test for adler32 (adler32-zlib) [ 2599.078156] Lustre: Lustre: Build Version: 2.17.57_1_g59aa375 [ 2599.304968] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 2600.951250] Key type lgssc registered [ 2602.007675] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2610.362254] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2610.390762] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2616.245610] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 2623.675143] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2624.091459] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2628.380709] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 2629.092697] LustreError: 86542:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2629.118590] LustreError: 86542:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2634.208662] LustreError: 86545:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2634.224616] LustreError: 86545:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2637.481632] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2637.819917] LustreError: 86542:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2637.861977] LustreError: 86542:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2637.870656] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2642.924807] LustreError: 86546:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2642.945984] LustreError: 86546:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2644.722225] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 2648.040537] LustreError: 87288:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2648.062355] LustreError: 87288:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2653.075796] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2653.088297] Lustre: MGS: Client d36df63a-2775-4891-a498-45e5c826053f (at 0@lo) reconnecting [ 2653.132241] LustreError: 86556:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2653.397490] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2653.472703] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2658.232785] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 2674.146109] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2674.158889] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2674.165364] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2674.192907] Lustre: Skipped 1 previous similar message [ 2679.270656] LustreError: 87269:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2679.290277] LustreError: 87269:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2679.404373] Lustre: server umount lustre-OST0000 complete [ 2684.384897] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2684.390105] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2684.403445] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2684.411819] Lustre: Skipped 2 previous similar messages [ 2689.505974] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2689.518902] Lustre: Skipped 1 previous similar message [ 2689.669773] Lustre: server umount lustre-MDT0000 complete [ 2693.534892] Lustre: server umount lustre-MDT0001 complete [ 2696.597907] Lustre: server umount MGS complete [ 2708.745960] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 18:27:26 (1786746446) [ 2738.893963] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_hostid [ 2747.257854] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 2781.750528] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2786.075605] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 2795.724846] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2795.924837] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2795.974856] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2796.086446] Lustre: lustre-MDT0001: new disk, initializing [ 2796.166324] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2796.191848] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2796.202809] Lustre: 93734:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2796.216867] LustreError: 93734:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2800.957346] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 2811.137316] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2811.344692] Lustre: lustre-OST0000: new disk, initializing [ 2811.347703] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2811.355815] Lustre: 94499:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2811.421799] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2817.746842] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 2819.684579] Lustre: 94905:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@00000000ef6ab451 x1873539328155392/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1786746574 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2826.708866] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2833.117932] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2833.149317] Lustre: MGS: Client c8c61dbc-49ea-4996-9502-9a1ce634bea3 (at 0@lo) reconnecting [ 2833.175060] LustreError: 93720:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2833.222524] Lustre: 93003:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2833.289430] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2833.351384] Lustre: lustre-MDT0000: new disk, initializing [ 2833.435554] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2833.457791] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2833.545150] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2834.529924] Lustre: 95568:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@0000000092432405 x1873539328162048/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1786746589 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2838.022693] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 2840.033355] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2843.626863] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2843.641399] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2843.724558] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2869.728551] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2869.740432] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2869.754838] Lustre: Skipped 1 previous similar message [ 2869.772612] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2872.290427] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2874.042450] Lustre: server umount lustre-OST0000 complete [ 2877.408846] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2877.414965] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2877.426816] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2877.431868] Lustre: Skipped 3 previous similar messages [ 2883.506692] Lustre: server umount lustre-MDT0000 complete [ 2893.243397] Lustre: server umount lustre-MDT0001 complete [ 2897.153058] Lustre: server umount MGS complete [ 2909.772842] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 18:30:47 (1786746647) [ 2927.737292] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 2930.422894] Key type lgssc unregistered [ 2930.721313] LNet: 98070:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2930.730163] LNetError: 98070:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2930.751564] LNet: Removed LNI 192.168.206.151@tcp [ 2931.564748] Key type .llcrypt unregistered [ 2931.567896] Key type ._llcrypt unregistered [ 2948.196668] Key type ._llcrypt registered [ 2948.199205] Key type .llcrypt registered [ 2948.289974] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 2949.228033] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2949.345079] alg: No test for adler32 (adler32-zlib) [ 2950.443506] Lustre: Lustre: Build Version: 2.17.57_1_g59aa375 [ 2950.686229] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 2952.311526] Key type lgssc registered [ 2953.476168] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2959.707933] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2959.720934] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2961.084536] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2964.882165] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 2971.628289] LustreError: 99294:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2971.659906] LustreError: 99294:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2973.191628] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2973.580887] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2977.645825] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 2984.388181] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2989.503079] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2994.775616] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2995.036235] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2999.601514] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 3005.503066] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3006.336656] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 3018.722385] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3018.727106] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3018.734458] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3021.794667] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3021.799180] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3021.799924] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3021.814915] Lustre: Skipped 1 previous similar message [ 3022.925435] Lustre: server umount lustre-OST0000 complete [ 3026.913259] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3026.917130] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3026.926385] Lustre: Skipped 1 previous similar message [ 3032.033597] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3032.040034] Lustre: Skipped 2 previous similar messages [ 3032.523884] Lustre: server umount lustre-MDT0000 complete [ 3035.818553] LustreError: 99273:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786746775 with bad export cookie 2546882156713180836 [ 3035.821181] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3035.827688] LustreError: 99273:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3036.200392] Lustre: server umount lustre-MDT0001 complete [ 3045.194597] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 3047.528452] Key type lgssc unregistered [ 3047.825201] LNet: 102830:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3047.837921] LNetError: 102830:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3047.868023] LNet: Removed LNI 192.168.206.151@tcp [ 3048.777291] Key type .llcrypt unregistered [ 3048.780460] Key type ._llcrypt unregistered [ 3062.839802] Key type ._llcrypt registered [ 3062.842503] Key type .llcrypt registered [ 3062.939116] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 3063.764425] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3063.820699] alg: No test for adler32 (adler32-zlib) [ 3064.897900] Lustre: Lustre: Build Version: 2.17.57_1_g59aa375 [ 3065.176882] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 3066.839156] Key type lgssc registered [ 3067.791541] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3072.504813] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3072.511832] LustreError: 103905:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3072.564390] LustreError: 103912:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3072.570934] LustreError: 103905:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4949 [ 3072.581081] LustreError: 103905:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3072.591811] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3072.594059] LustreError: 103905:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3074.116626] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3081.029532] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3081.270431] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3085.154475] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 3090.448725] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing lsmod [ 3093.273187] LustreError: 105115:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3093.294741] LustreError: 105118:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3093.299842] LustreError: 105115:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4980 [ 3093.304648] LustreError: 105115:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3093.312388] LustreError: 105115:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3093.320585] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3093.638346] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3095.998152] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3096.014569] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3096.025666] Lustre: Skipped 2 previous similar messages [ 3101.159823] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3101.166238] Lustre: Skipped 2 previous similar messages [ 3106.274280] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3106.285884] Lustre: Skipped 2 previous similar messages [ 3110.367160] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3110.686261] Lustre: server umount lustre-MDT0000 complete [ 3111.399369] LustreError: 103946:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3111.435226] LustreError: 103946:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 3114.094618] LustreError: 103927:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786746853 with bad export cookie 7260140087676954625 [ 3114.101391] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3114.109737] LustreError: 103927:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3114.487263] LustreError: 104380:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000008dadc61d x1873539816771712/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3114.640364] Lustre: server umount lustre-MDT0001 complete [ 3128.221470] Lustre: server umount lustre-OST0000 complete [ 3135.893312] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3136.305027] LustreError: 106183:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3136.384413] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3141.043426] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 3141.605253] LustreError: 106184:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3146.723817] LustreError: 106183:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3148.732117] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3149.051360] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3153.221699] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 3160.176872] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3166.495818] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3175.813052] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3176.374933] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3181.495234] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3182.932417] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 3190.803376] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3204.064286] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3204.071336] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3204.079836] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3204.110633] Lustre: Skipped 3 previous similar messages [ 3209.200296] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3209.214398] Lustre: Skipped 1 previous similar message [ 3209.824675] Lustre: server umount lustre-OST0000 complete [ 3214.308485] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3214.334386] Lustre: Skipped 1 previous similar message [ 3219.425919] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3219.435236] Lustre: Skipped 4 previous similar messages [ 3219.795358] Lustre: server umount lustre-MDT0000 complete [ 3223.475189] LustreError: 106166:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786746962 with bad export cookie 7260140087676955311 [ 3223.491490] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3223.509499] LustreError: 106166:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3223.897419] Lustre: server umount lustre-MDT0001 complete [ 3235.227610] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 3237.742272] Key type lgssc unregistered [ 3238.034243] LNet: 109738:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3238.043344] LNetError: 109738:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3238.061329] LNet: Removed LNI 192.168.206.151@tcp [ 3238.869395] Key type .llcrypt unregistered [ 3238.871566] Key type ._llcrypt unregistered [ 3261.322434] Key type ._llcrypt registered [ 3261.325279] Key type .llcrypt registered [ 3261.420960] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 3262.249369] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3262.262278] alg: No test for adler32 (adler32-zlib) [ 3263.365765] Lustre: Lustre: Build Version: 2.17.57_1_g59aa375 [ 3263.578820] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 3265.271163] Key type lgssc registered [ 3266.328926] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3274.199533] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3274.235478] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3275.880646] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3280.785284] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 3286.502946] LustreError: 111154:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3286.521044] LustreError: 111154:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3290.953944] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3291.343977] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3295.924502] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 3303.794474] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3310.673931] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3317.509290] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3317.934706] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3321.037909] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 3325.663974] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 3333.831984] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3342.139662] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3345.783929] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3352.033427] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3352.038978] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3352.151918] Lustre: server umount lustre-OST0000 complete [ 3357.154831] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3357.169263] Lustre: Skipped 1 previous similar message [ 3357.174513] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3357.183294] Lustre: Skipped 1 previous similar message [ 3362.272410] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3362.275216] Lustre: Skipped 1 previous similar message [ 3362.719358] Lustre: server umount lustre-MDT0000 complete [ 3363.306344] LustreError: 111150:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3363.325582] LustreError: 111150:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3366.132861] LustreError: 111134:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786747105 with bad export cookie 2319597261294937850 [ 3366.140302] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3366.149312] LustreError: 111134:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3366.736588] Lustre: server umount lustre-MDT0001 complete [ 3377.110567] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 3379.551695] Key type lgssc unregistered [ 3379.805101] LNet: 114795:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3379.815175] LNetError: 114795:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3379.831860] LNet: Removed LNI 192.168.206.151@tcp [ 3380.494745] Key type .llcrypt unregistered [ 3380.497984] Key type ._llcrypt unregistered [ 3396.913923] Key type ._llcrypt registered [ 3396.915512] Key type .llcrypt registered [ 3397.006684] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 3397.849240] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3397.974172] alg: No test for adler32 (adler32-zlib) [ 3399.018483] Lustre: Lustre: Build Version: 2.17.57_1_g59aa375 [ 3399.249890] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 3400.904102] Key type lgssc registered [ 3401.837529] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3408.738388] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3408.755422] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3410.094210] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3414.149547] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 3420.642654] LustreError: 116018:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3420.654291] LustreError: 116018:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3421.313949] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3421.737695] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3425.244108] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 3431.396054] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3436.652178] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3443.454273] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3443.769255] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3449.304276] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 3455.029586] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3456.338400] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3465.189855] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3465.190714] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3465.194491] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3465.214639] Lustre: Skipped 1 previous similar message [ 3469.599804] Lustre: server umount lustre-OST0000 complete [ 3472.866638] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3472.872953] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3472.883229] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3472.887329] Lustre: Skipped 1 previous similar message [ 3475.424418] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3475.425031] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3480.546614] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3480.558291] Lustre: Skipped 2 previous similar messages [ 3485.667990] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3485.675346] Lustre: Skipped 1 previous similar message [ 3487.199143] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3487.338812] Lustre: server umount lustre-MDT0000 complete [ 3490.625675] LustreError: 115999:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786747229 with bad export cookie 9305257672328884663 [ 3490.642982] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3490.978609] Lustre: server umount lustre-MDT0001 complete [ 3501.798743] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 3504.322497] Key type lgssc unregistered [ 3504.617666] LNet: 119550:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3504.626886] LNetError: 119550:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3504.649215] LNet: Removed LNI 192.168.206.151@tcp [ 3505.403236] Key type .llcrypt unregistered [ 3505.406227] Key type ._llcrypt unregistered [ 3530.580772] Key type ._llcrypt registered [ 3530.584147] Key type .llcrypt registered [ 3530.714873] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 3532.142263] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3532.229742] alg: No test for adler32 (adler32-zlib) [ 3533.236085] Lustre: Lustre: Build Version: 2.17.57_1_g59aa375 [ 3533.494320] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 3535.143165] Key type lgssc registered [ 3536.085577] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3542.227944] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3542.243881] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3543.541265] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3546.670883] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 3553.725283] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3553.767253] LustreError: 120968:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3553.789468] LustreError: 120968:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3554.050282] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3557.917705] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 3564.800543] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3569.654714] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3575.677134] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3576.091562] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3581.635692] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 3588.088496] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3589.438920] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3594.856953] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3598.098695] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3598.103255] Lustre: Skipped 2 previous similar messages [ 3600.931276] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3600.943794] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3601.949807] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3601.955665] Lustre: Skipped 2 previous similar messages [ 3607.610780] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3607.614053] Lustre: Skipped 1 previous similar message [ 3611.665637] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.206.51@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3611.678119] Lustre: Skipped 2 previous similar messages [ 3616.292790] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.206.51@tcp 192.168.0.[2,10]@tcp [ 3621.036797] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.206.51/24@tcp in log lustre-MDT0000 [ 3621.045746] Lustre: Skipped 2 previous similar messages [ 3624.435089] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.206.51/24@tcp [ 3624.438536] Lustre: Skipped 1 previous similar message [ 3635.175555] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3635.177288] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3635.183902] LustreError: Skipped 1 previous similar message [ 3635.207499] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3635.684176] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3638.942749] Lustre: server umount lustre-OST0000 complete [ 3645.921020] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3645.935902] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3645.938515] Lustre: Skipped 1 previous similar message [ 3648.842835] Lustre: server umount lustre-MDT0000 complete [ 3651.040958] LustreError: 120968:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3651.059132] LustreError: 120968:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3652.379672] LustreError: 120948:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786747391 with bad export cookie 2971922144247450397 [ 3652.391792] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3652.394548] LustreError: 120948:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3652.754809] Lustre: server umount lustre-MDT0001 complete [ 3662.984082] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 3665.549392] Key type lgssc unregistered [ 3665.826913] LNet: 125915:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3665.833979] LNetError: 125915:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3665.855413] LNet: Removed LNI 192.168.206.151@tcp [ 3666.612052] Key type .llcrypt unregistered [ 3666.618492] Key type ._llcrypt unregistered [ 3692.285716] Key type ._llcrypt registered [ 3692.288175] Key type .llcrypt registered [ 3692.370853] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 3693.096659] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3693.134738] alg: No test for adler32 (adler32-zlib) [ 3694.164537] Lustre: Lustre: Build Version: 2.17.57_1_g59aa375 [ 3694.413047] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 3696.087173] Key type lgssc registered [ 3697.219785] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3704.264400] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3704.287216] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3705.597111] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3705.609062] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3705.624899] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.206.51/24@tcp [ 3709.227622] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 3715.043128] LustreError: 127477:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3715.058441] LustreError: 127477:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3716.286290] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3716.670871] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3716.689827] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.206.51/24@tcp [ 3716.735057] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3720.296481] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 3726.685540] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3732.237219] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3738.097293] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3738.320291] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3743.596795] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 3750.236280] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3750.651674] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 3761.124652] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3761.125385] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3761.137836] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3761.147274] Lustre: Skipped 1 previous similar message [ 3765.267884] Lustre: server umount lustre-OST0000 complete [ 3771.361766] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3771.363249] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3771.369840] Lustre: Skipped 2 previous similar messages [ 3774.525930] Lustre: server umount lustre-MDT0000 complete [ 3776.480000] LustreError: 128222:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3776.514773] LustreError: 128222:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3777.631122] LustreError: 129324:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786747516 with bad export cookie 10922678710718441967 [ 3777.638469] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3777.642701] LustreError: 129324:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3777.934262] Lustre: server umount lustre-MDT0001 complete [ 3787.713801] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 3789.849926] Key type lgssc unregistered [ 3790.072723] LNet: 131068:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3790.082262] LNetError: 131068:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3790.097618] LNet: Removed LNI 192.168.206.151@tcp [ 3790.794456] Key type .llcrypt unregistered [ 3790.798321] Key type ._llcrypt unregistered [ 3811.943611] Key type ._llcrypt registered [ 3811.947292] Key type .llcrypt registered [ 3812.075539] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_hostid [ 3824.271147] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 3825.268443] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3825.304453] alg: No test for adler32 (adler32-zlib) [ 3826.469530] Lustre: Lustre: Build Version: 2.17.57_1_g59aa375 [ 3826.658244] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 3828.314605] Key type lgssc registered [ 3829.439897] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3860.867184] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3860.893360] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3862.052914] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3862.077847] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3862.160835] Lustre: lustre-MDT0000: new disk, initializing [ 3862.239600] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3862.261264] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3865.751395] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 3874.233608] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3874.352901] Lustre: 134446:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3874.399693] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3874.402945] Lustre: Skipped 1 previous similar message [ 3874.466364] Lustre: lustre-MDT0001: new disk, initializing [ 3874.546179] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3874.583872] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3874.597929] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3877.996533] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 3885.462492] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3889.687857] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3894.874779] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3895.082888] Lustre: lustre-OST0000: new disk, initializing [ 3895.085914] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3895.091745] Lustre: 136414:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3895.177719] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3900.246645] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 3904.071483] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3904.079906] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3904.187537] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3908.136517] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3911.935333] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3912.169857] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3916.375558] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3916.553198] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3919.843456] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3919.849359] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3919.858783] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3919.870840] Lustre: Skipped 1 previous similar message [ 3924.367665] Lustre: server umount lustre-OST0000 complete [ 3930.081973] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3930.089646] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3930.114031] Lustre: Skipped 1 previous similar message [ 3930.135305] Lustre: Skipped 2 previous similar messages [ 3934.987082] Lustre: server umount lustre-MDT0000 complete [ 3938.596229] LustreError: 135241:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786747677 with bad export cookie 15059609680567007574 [ 3938.603148] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3938.615235] LustreError: 135241:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3938.992443] Lustre: server umount lustre-MDT0001 complete [ 3947.065327] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3947.436306] LustreError: 138631:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3947.448174] LustreError: 138631:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 3947.480611] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3951.513961] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 3952.616740] LustreError: 138632:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3957.730260] LustreError: 138631:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3958.585409] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3958.878163] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3962.560504] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 3968.947420] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3973.769830] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3979.269835] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3979.551806] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3984.575288] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 3991.435908] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3995.506914] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3995.672278] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4000.620212] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4000.839130] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4008.202631] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4008.360711] Lustre: lustre-OST0001: new disk, initializing [ 4008.365162] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4008.370089] Lustre: 141999:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4008.461532] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4014.307956] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 4016.753242] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4016.766224] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4016.858119] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4021.536851] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4021.768290] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4025.986954] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4026.166535] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4031.374411] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff98478635e800.ost_server_uuid 50 [ 4032.728224] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff98478635e800.ost_server_uuid in FULL state after 0 sec [ 4042.722193] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4042.728864] LustreError: Skipped 1 previous similar message [ 4042.729788] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4042.734802] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4042.742242] Lustre: Skipped 1 previous similar message [ 4042.752591] Lustre: Skipped 1 previous similar message [ 4045.361883] Lustre: server umount lustre-OST0001 complete [ 4047.856778] LustreError: 140451:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4047.887743] LustreError: 140451:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4051.428200] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4051.435408] LustreError: Skipped 1 previous similar message [ 4051.439388] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4051.461846] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4052.963182] LustreError: 142461:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4052.986443] LustreError: 142461:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4054.804312] Lustre: server umount lustre-OST0000 complete [ 4061.155416] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4061.167087] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4061.182331] Lustre: Skipped 1 previous similar message [ 4061.188663] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4061.192111] Lustre: Skipped 2 previous similar messages [ 4064.581539] Lustre: server umount lustre-MDT0000 complete [ 4068.104247] LustreError: 138613:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786747807 with bad export cookie 15059609680567008456 [ 4068.106691] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4068.114661] LustreError: 138613:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4068.382945] Lustre: server umount lustre-MDT0001 complete [ 4085.781477] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 4088.153624] Key type lgssc unregistered [ 4088.466375] LNet: 145004:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4088.471630] LNetError: 145004:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4088.493662] LNet: Removed LNI 192.168.206.151@tcp [ 4089.157793] Key type .llcrypt unregistered [ 4089.160749] Key type ._llcrypt unregistered [ 4127.905844] Key type ._llcrypt registered [ 4127.915332] Key type .llcrypt registered [ 4128.086817] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_hostid [ 4144.732112] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 4145.970201] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4145.991793] alg: No test for adler32 (adler32-zlib) [ 4147.059910] Lustre: Lustre: Build Version: 2.17.57_1_g59aa375 [ 4147.289589] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 4148.975226] Key type lgssc registered [ 4149.940625] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4180.204926] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4180.251643] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4181.411649] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4181.440989] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4181.545110] Lustre: lustre-MDT0000: new disk, initializing [ 4181.671591] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4181.693726] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4185.976194] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 4196.031176] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4196.244381] Lustre: 149155:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4196.312399] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4196.318443] Lustre: Skipped 1 previous similar message [ 4196.378898] Lustre: lustre-MDT0001: new disk, initializing [ 4196.475349] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4196.504818] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4196.522900] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4200.734713] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 4209.718093] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4215.429409] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4220.989988] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4221.302590] Lustre: lustre-OST0000: new disk, initializing [ 4221.306098] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4221.322161] Lustre: 151124:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4221.446276] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4227.307782] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 4227.607775] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4227.615893] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4227.691243] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4235.748447] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4240.286447] Lustre: Failing over lustre-OST0000 [ 4240.383826] Lustre: server umount lustre-OST0000 complete [ 4258.209793] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4258.640907] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4258.663708] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4259.208352] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4259.279325] Lustre: 147383:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786747982/real 1786747982] req@0000000001216d49 x1873540951343616/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786747998 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4259.295250] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4259.298818] Lustre: 147383:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4259.361304] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4259.386386] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 4264.665392] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 4265.440254] Lustre: 147380:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786747987/real 1786747987] req@000000003d9b7f66 x1873540951344896/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786748003 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4265.459508] Lustre: 147380:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4268.245656] Lustre: Failing over lustre-MDT0000 [ 4268.519202] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4268.524515] Lustre: server umount lustre-MDT0000 complete [ 4268.534359] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4268.567627] Lustre: Skipped 1 previous similar message [ 4269.280093] Lustre: 147381:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786747992/real 1786747992] req@00000000c99904f7 x1873540951346048/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786748008 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4269.540637] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4274.528949] LustreError: 150887:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4274.550282] LustreError: 150887:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 4274.659040] Lustre: 147383:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786747997/real 1786747997] req@000000002a67edfd x1873540951347200/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786748013 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4279.656608] LustreError: 149163:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4279.678996] LustreError: 149163:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4284.774141] LustreError: 149163:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4284.795085] LustreError: 149163:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4285.407131] Lustre: 149160:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786748008/real 1786748008] req@0000000017363a32 x1873540951355520/t0(0) o101->MGC192.168.206.151@tcp@0@lo:26/25 lens 328/344 e 0 to 1 dl 1786748024 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4285.450485] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4285.466561] Lustre: 149160:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.206.151@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 4287.014666] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4290.529092] LustreError: 149165:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4290.543525] LustreError: 149165:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 4295.648057] LustreError: 147379:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000028549e0 x1873540951361408/t0(0) o250->MGC192.168.206.151@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4295.649310] LustreError: 149169:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4295.685165] LustreError: 149169:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4295.908769] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4295.964952] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4296.053972] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4299.603608] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 4301.293749] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4301.298384] Lustre: Skipped 1 previous similar message [ 4301.318369] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 4301.370710] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4306.912661] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4306.921091] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4306.944090] Lustre: Skipped 1 previous similar message [ 4306.959141] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4310.497016] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4310.504063] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4311.525398] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4311.530321] Lustre: Skipped 1 previous similar message [ 4312.693932] Lustre: server umount lustre-OST0000 complete [ 4316.644074] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4316.646132] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4316.664562] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4316.676699] Lustre: Skipped 2 previous similar messages [ 4316.685247] Lustre: Skipped 1 previous similar message [ 4321.762331] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4321.768747] Lustre: Skipped 1 previous similar message [ 4322.364894] Lustre: server umount lustre-MDT0000 complete [ 4325.740658] LustreError: 149148:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786748064 with bad export cookie 14397606260995026631 [ 4325.743652] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4325.754350] LustreError: 149148:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4326.080968] Lustre: server umount lustre-MDT0001 complete [ 4336.710332] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 4339.084370] Key type lgssc unregistered [ 4339.323860] LNet: 154788:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4339.330343] LNetError: 154788:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4339.345482] LNet: Removed LNI 192.168.206.151@tcp [ 4340.131723] Key type .llcrypt unregistered [ 4340.139124] Key type ._llcrypt unregistered [ 4374.152792] Key type ._llcrypt registered [ 4374.156079] Key type .llcrypt registered [ 4374.257757] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 4375.403265] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4375.474449] alg: No test for adler32 (adler32-zlib) [ 4376.535775] Lustre: Lustre: Build Version: 2.17.57_1_g59aa375 [ 4376.814680] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 4378.520371] Key type lgssc registered [ 4379.506639] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4386.351167] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4386.376305] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4387.841191] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4392.184406] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 4398.051534] LustreError: 156309:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4398.063326] LustreError: 156309:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4400.752504] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4403.173595] LustreError: 156310:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4404.619593] LustreError: 156338:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4404.631586] LustreError: 156338:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4404.640720] LustreError: 156338:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 16, retries 0, failed: rc = -5 [ 4405.022565] Lustre: server umount lustre-MDT0000 complete [ 4421.137689] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_hostid [ 4430.721775] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 4466.456523] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4466.649206] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4466.671285] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4466.741863] Lustre: lustre-MDT0000: new disk, initializing [ 4466.803485] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4466.821346] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4471.318263] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 4482.524145] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4482.736422] Lustre: 159642:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4482.819188] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4482.823856] Lustre: Skipped 1 previous similar message [ 4482.939793] Lustre: lustre-MDT0001: new disk, initializing [ 4483.089400] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4483.139232] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4483.149769] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4488.382840] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 4498.024319] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4504.498648] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4510.871771] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4511.139107] Lustre: lustre-OST0000: new disk, initializing [ 4511.145386] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4511.153778] Lustre: 161609:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4511.236388] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4517.330991] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 4518.488091] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4518.499254] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4518.564623] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4525.186489] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4528.406613] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4529.876658] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 18:57:47 (1786748267) [ 4774.881818] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4774.890498] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4774.906792] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4777.951736] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4777.953425] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4777.981032] Lustre: Skipped 1 previous similar message [ 4777.988328] Lustre: Skipped 2 previous similar messages [ 4781.001943] Lustre: server umount lustre-MDT0000 complete [ 4783.072986] LustreError: 159652:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4783.092601] LustreError: 159652:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4784.890418] LustreError: 160447:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786748524 with bad export cookie 9808861358923877012 [ 4784.890841] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4784.902051] LustreError: 160447:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4788.195656] LustreError: 160445:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4788.197900] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4788.233494] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4791.445397] Lustre: server umount lustre-MDT0001 complete [ 4802.323734] Lustre: server umount lustre-OST0000 complete [ 4814.947926] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_hostid [ 4824.465440] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 4861.555850] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4861.903323] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4861.944734] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4862.032000] Lustre: lustre-MDT0000: new disk, initializing [ 4862.136975] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4862.167082] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4867.149271] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 4878.028739] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4878.130733] Lustre: 166285:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4878.170057] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4878.172647] Lustre: Skipped 1 previous similar message [ 4878.243440] Lustre: lustre-MDT0001: new disk, initializing [ 4878.311965] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4878.336111] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4878.352780] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4882.985784] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 4892.221972] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4898.273933] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4906.067476] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4906.367675] Lustre: lustre-OST0000: new disk, initializing [ 4906.372773] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4906.379319] Lustre: 168253:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4906.450674] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4912.216803] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4912.234294] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4912.348579] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4912.799112] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 4921.865178] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4925.917909] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4926.167777] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4931.219478] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4931.467228] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4938.218719] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4938.223227] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4938.229670] Lustre: Skipped 1 previous similar message [ 4938.231170] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4938.237518] Lustre: Skipped 2 previous similar messages [ 4939.702239] Lustre: server umount lustre-OST0000 complete [ 4943.329795] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4943.344120] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4949.227556] Lustre: server umount lustre-MDT0000 complete [ 4952.873130] LustreError: 166276:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786748692 with bad export cookie 9808861358924035835 [ 4952.878698] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4952.889725] LustreError: 166276:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4953.304511] Lustre: server umount lustre-MDT0001 complete [ 4962.808040] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 19:05:00 (1786748700) [ 4980.059961] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_hostid [ 4988.773746] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 5027.919623] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5028.137133] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5028.179701] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5028.257574] Lustre: lustre-MDT0000: new disk, initializing [ 5028.335748] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5028.358277] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5033.196863] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 5044.936564] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5045.050347] Lustre: 173012:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5045.465728] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5050.913778] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 5061.481051] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5067.817705] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5074.603757] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5074.806869] Lustre: lustre-OST0000: new disk, initializing [ 5074.809335] Lustre: Skipped 1 previous similar message [ 5074.812545] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5074.819904] Lustre: Skipped 2 previous similar messages [ 5074.830162] Lustre: 174987:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5081.316988] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 5084.772039] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5084.779813] Lustre: Skipped 1 previous similar message [ 5084.784518] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5084.891107] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5090.521944] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5100.535396] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5100.546827] LustreError: Skipped 1 previous similar message [ 5100.550981] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5100.567970] Lustre: Skipped 1 previous similar message [ 5100.578942] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5100.586082] Lustre: Skipped 4 previous similar messages [ 5103.894950] Lustre: server umount lustre-OST0000 complete [ 5114.160915] Lustre: server umount lustre-MDT0000 complete [ 5115.872861] LustreError: 174577:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5115.900196] LustreError: 174577:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5118.283634] LustreError: 173820:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786748857 with bad export cookie 9808861358924036724 [ 5118.289379] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5118.311762] LustreError: 173820:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5118.764912] Lustre: server umount lustre-MDT0001 complete [ 5129.086060] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 19:07:46 (1786748866) [ 5144.436867] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_hostid [ 5153.067097] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 5191.715459] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5192.228738] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5192.283599] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5192.384191] Lustre: lustre-MDT0000: new disk, initializing [ 5192.461694] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5192.469661] Lustre: Skipped 2 previous similar messages [ 5192.486815] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5198.647741] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 5212.266135] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5212.345855] Lustre: 179311:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5212.548339] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5219.416902] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 5233.008426] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5240.216728] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5249.183498] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5249.451861] Lustre: 181294:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5255.713452] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5255.825906] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5257.702395] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 5268.286225] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5281.250465] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5281.275239] LustreError: Skipped 1 previous similar message [ 5281.278499] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5281.297576] Lustre: Skipped 3 previous similar messages [ 5281.316070] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5281.332506] Lustre: Skipped 2 previous similar messages [ 5283.607471] Lustre: server umount lustre-OST0000 complete [ 5289.440642] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5289.452594] LustreError: Skipped 1 previous similar message [ 5294.550974] Lustre: server umount lustre-MDT0000 complete [ 5297.120786] LustreError: 179319:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5297.172786] LustreError: 179319:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5299.179212] LustreError: 179305:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786749038 with bad export cookie 9808861358924037809 [ 5299.199423] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5299.202303] LustreError: 179305:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5299.663621] Lustre: server umount lustre-MDT0001 complete [ 5317.450876] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 5321.103973] Key type lgssc unregistered [ 5321.538795] LNet: 183166:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5321.550879] LNetError: 183166:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5321.571653] LNet: Removed LNI 192.168.206.151@tcp [ 5322.512116] Key type .llcrypt unregistered [ 5322.515203] Key type ._llcrypt unregistered [ 5350.948251] Key type ._llcrypt registered [ 5350.954769] Key type .llcrypt registered [ 5351.130756] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 5352.454311] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5352.576202] alg: No test for adler32 (adler32-zlib) [ 5353.690331] Lustre: Lustre: Build Version: 2.17.57_1_g59aa375 [ 5354.028500] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 5355.735157] Key type lgssc registered [ 5357.343540] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5366.366128] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5366.388423] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5367.926904] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5373.192838] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 5378.532266] LustreError: 184583:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5378.575149] LustreError: 184583:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5382.870830] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5383.250343] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5388.034621] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 5395.578663] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5401.078315] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5408.452937] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5408.739432] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 5408.854788] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5412.991117] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5415.373125] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 5423.067452] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5444.066611] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5444.078893] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5444.083650] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5444.098349] Lustre: Skipped 1 previous similar message [ 5447.212353] Lustre: server umount lustre-OST0000 complete [ 5454.308607] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5454.315856] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5454.318777] Lustre: Skipped 1 previous similar message [ 5454.329631] Lustre: Skipped 2 previous similar messages [ 5457.243459] Lustre: server umount lustre-MDT0000 complete [ 5459.425017] LustreError: 184584:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5459.469210] LustreError: 184584:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5460.838791] LustreError: 184564:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786749200 with bad export cookie 142766383794806427 [ 5460.842713] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5460.852469] LustreError: 184564:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5461.124708] Lustre: server umount lustre-MDT0001 complete [ 5471.463797] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 5473.622635] Key type lgssc unregistered [ 5473.907071] LNet: 188141:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5473.919238] LNetError: 188141:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5473.936365] LNet: Removed LNI 192.168.206.151@tcp [ 5474.763320] Key type .llcrypt unregistered [ 5474.767384] Key type ._llcrypt unregistered [ 5501.132216] Key type ._llcrypt registered [ 5501.134403] Key type .llcrypt registered [ 5501.232598] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 5502.201076] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5502.223088] alg: No test for adler32 (adler32-zlib) [ 5503.486764] Lustre: Lustre: Build Version: 2.17.57_1_g59aa375 [ 5503.950268] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 5505.719164] Key type lgssc registered [ 5508.183524] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5519.742732] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5519.801727] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5521.339294] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5527.044495] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 5531.617617] LustreError: 189564:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5531.660387] LustreError: 189564:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5535.498038] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5535.966370] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5540.123948] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 5547.739164] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5554.086476] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5560.483343] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5560.870733] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5567.006052] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5567.178934] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 5575.548745] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5582.715337] Lustre: lustre-OST0000: Not available for connect from 192.168.206.51@tcp (stopping) [ 5586.403214] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5586.417714] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5586.454518] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5587.437443] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5587.454633] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5587.461528] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5587.461536] Lustre: Skipped 1 previous similar message [ 5588.832502] Lustre: server umount lustre-OST0000 complete [ 5598.328542] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff9847859b3800.ost_server_uuid 50 [ 5600.712274] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5617.125603] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5617.163449] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5617.171323] Lustre: Skipped 2 previous similar messages [ 5620.500337] Lustre: server umount lustre-MDT0000 complete [ 5622.260495] LustreError: 189564:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5622.280107] LustreError: 189564:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5624.854354] LustreError: 189545:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786749364 with bad export cookie 105567840114729833 [ 5624.862861] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5624.882430] LustreError: 189545:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5625.397193] Lustre: server umount lustre-MDT0001 complete [ 5634.772294] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 19:16:12 (1786749372) [ 5640.604835] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5641.109424] LustreError: 193476:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5641.276350] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5645.685583] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 5646.308733] LustreError: 193477:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5651.426636] LustreError: 193476:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5654.592653] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5655.243893] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5659.635627] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 5668.185747] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5674.453985] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5683.024621] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5683.657776] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5686.732802] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5691.365916] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 5700.820435] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5710.174053] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5710.659942] Lustre: lustre-OST0001: new disk, initializing [ 5710.663807] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5710.678456] Lustre: 196320:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5710.916159] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5716.251887] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5716.281326] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5716.477928] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5720.198893] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 5731.921722] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5736.935177] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5736.939342] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5736.954832] Lustre: Skipped 1 previous similar message [ 5736.955885] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5741.413533] Lustre: server umount lustre-OST0000 complete [ 5742.056075] LustreError: 195298:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5742.076778] LustreError: 195298:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5747.290413] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5747.548572] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5751.499710] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5751.728384] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5758.305144] LustreError: 195297:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.206.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5758.324620] LustreError: 195297:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 5767.656523] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5767.665607] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5767.685140] Lustre: Skipped 1 previous similar message [ 5767.689588] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5767.696843] Lustre: Skipped 3 previous similar messages [ 5771.924588] Lustre: server umount lustre-OST0001 complete [ 5777.891760] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5777.896930] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5777.919111] Lustre: Skipped 1 previous similar message [ 5777.928374] LustreError: Skipped 1 previous similar message [ 5782.401984] Lustre: server umount lustre-MDT0000 complete [ 5786.381673] LustreError: 193458:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786749525 with bad export cookie 105567840114730904 [ 5786.388805] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5786.392301] LustreError: 193458:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5786.846097] Lustre: server umount lustre-MDT0001 complete [ 5810.822327] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 19:19:08 (1786749548) [ 5817.677376] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5817.979180] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5818.026577] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5818.241616] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5822.521643] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 5831.783966] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5831.956095] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5832.015455] Lustre: 199758:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5837.419251] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 5845.881681] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5851.536459] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5858.653478] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5858.775091] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5865.815558] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 5868.709788] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 5875.909942] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5884.134436] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5884.294320] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5884.597880] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5884.603491] Lustre: Skipped 2 previous similar messages [ 5892.055213] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 5893.330188] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 5900.326823] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5903.849877] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5903.874205] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5903.887469] Lustre: Skipped 3 previous similar messages [ 5909.501564] Lustre: server umount lustre-OST0000 complete [ 5911.398457] LustreError: 202100:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.206.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5911.420168] LustreError: 202100:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 5929.440135] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5931.373364] Lustre: server umount lustre-OST0001 complete [ 5939.681812] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5939.685113] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5939.712658] Lustre: Skipped 4 previous similar messages [ 5939.724274] Lustre: Skipped 6 previous similar messages [ 5941.489359] Lustre: server umount lustre-MDT0000 complete [ 5945.287717] LustreError: 199749:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786749684 with bad export cookie 105567840114732311 [ 5945.298179] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5945.302354] LustreError: 199749:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5945.635541] Lustre: server umount lustre-MDT0001 complete [ 5969.728787] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 19:21:47 (1786749707) [ 5989.055756] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_hostid [ 5998.893030] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 6034.880823] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6035.110992] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6035.132241] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6035.193170] Lustre: lustre-MDT0000: new disk, initializing [ 6035.252837] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6035.267516] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6039.018360] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 6048.757498] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6048.871774] Lustre: 207858:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6048.927894] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6048.930794] Lustre: Skipped 1 previous similar message [ 6049.012321] Lustre: lustre-MDT0001: new disk, initializing [ 6049.113865] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6049.126960] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6053.861385] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 6064.664707] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6073.304182] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6081.615973] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6081.903274] Lustre: lustre-OST0000: new disk, initializing [ 6081.908221] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6081.916586] Lustre: 209827:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6090.083248] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 6090.828909] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6090.836289] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6090.963677] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6100.273645] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6105.533638] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6105.795443] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6111.270553] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6111.459861] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6116.832725] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6116.839866] LustreError: Skipped 2 previous similar messages [ 6116.842277] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6116.846093] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6116.864130] Lustre: Skipped 1 previous similar message [ 6128.105374] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6128.308090] Lustre: server umount lustre-OST0000 complete [ 6138.544187] Lustre: server umount lustre-MDT0000 complete [ 6142.437424] LustreError: 207868:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6142.453061] LustreError: 207868:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 13 previous similar messages [ 6142.473726] LustreError: 207850:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786749881 with bad export cookie 105567840114734096 [ 6142.480391] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6142.492727] LustreError: 207850:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6142.900316] Lustre: server umount lustre-MDT0001 complete [ 6150.827821] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6156.252687] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 6163.752435] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6168.042762] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 6175.605887] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6181.673429] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6188.565727] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6195.451500] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 6203.696892] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6208.753472] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6208.987582] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6213.860876] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6214.112296] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6216.672201] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6216.691553] LustreError: Skipped 1 previous similar message [ 6222.377283] Lustre: server umount lustre-OST0000 complete [ 6229.114388] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6234.112872] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6263.664690] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6269.943504] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 6272.232228] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6272.270686] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6272.272323] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6272.309151] Lustre: Skipped 1 previous similar message [ 6277.746239] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6280.707926] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 1 sec [ 6287.840340] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6287.851433] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6287.855265] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6287.864933] Lustre: Skipped 5 previous similar messages [ 6287.870763] Lustre: Skipped 13 previous similar messages [ 6291.228733] Lustre: server umount lustre-OST0000 complete [ 6309.344231] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6309.556474] Lustre: server umount lustre-MDT0000 complete [ 6313.443741] LustreError: 213939:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6313.470178] LustreError: 213939:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 6313.623990] LustreError: 212023:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786750052 with bad export cookie 105567840114734978 [ 6313.628785] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6313.635903] LustreError: 212023:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6314.040514] Lustre: server umount lustre-MDT0001 complete [ 6322.603642] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 19:27:40 (1786750060) [ 6329.455031] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6330.149473] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6330.157626] Lustre: Skipped 6 previous similar messages [ 6334.917598] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 6344.288556] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6350.073173] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 6358.186330] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6365.684784] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6374.150489] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6381.904302] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 6390.147743] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6395.592684] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6395.785434] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6400.610874] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6400.839666] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6408.558115] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6408.776593] Lustre: lustre-OST0001: new disk, initializing [ 6408.786072] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6408.791926] Lustre: 221386:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6415.441402] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 6416.512219] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6416.527456] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6416.687423] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6426.198863] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6432.021824] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6432.313569] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6437.115983] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6437.424340] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6442.479257] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6442.499370] LustreError: Skipped 2 previous similar messages [ 6445.914336] Lustre: server umount lustre-OST0001 complete [ 6453.416400] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6453.724434] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6459.161252] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6459.378750] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6490.554488] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6496.268149] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6496.293282] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6496.295098] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6500.469405] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 6509.896238] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6511.799324] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6520.516434] Lustre: server umount lustre-OST0001 complete [ 6544.683310] Lustre: server umount lustre-OST0000 complete [ 6549.476243] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6549.487571] Lustre: Skipped 8 previous similar messages [ 6549.490595] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6549.498699] Lustre: Skipped 14 previous similar messages [ 6563.295307] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6563.565169] Lustre: server umount lustre-MDT0000 complete [ 6567.131027] LustreError: 217994:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786750306 with bad export cookie 105567840114735944 [ 6567.137809] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6567.141563] LustreError: 217994:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6573.538323] Lustre: server umount lustre-MDT0001 complete [ 6598.663429] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 19:32:16 (1786750336) [ 6605.528417] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6605.907943] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6605.967520] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6610.552748] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 6618.938801] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6618.994851] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6619.012708] Lustre: 226629:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6623.831293] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 6631.281657] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6637.278647] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6645.125557] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6645.305212] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6651.501565] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 6660.429818] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6669.828084] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6669.932112] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6675.584204] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 6682.256391] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6684.029749] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6688.512735] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6688.737714] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6692.782493] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6693.021729] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6698.275496] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff98479098f000.ost_server_uuid 50 [ 6700.019919] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff98479098f000.ost_server_uuid in IDLE state after 0 sec [ 6701.081523] Lustre: Permanently deactivating lustre-OST0001 [ 6701.103436] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6705.309855] Lustre: Permanently reactivating lustre-OST0001 [ 6705.335775] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 6705.338537] Lustre: Skipped 2 previous similar messages [ 6710.764639] LustreError: lustre-OST0001-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 6710.771468] LustreError: Skipped 4 previous similar messages [ 6715.750454] Lustre: server umount lustre-OST0001 complete [ 6717.930954] LustreError: 228475:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6717.942857] LustreError: 228475:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 40 previous similar messages [ 6725.414510] Lustre: server umount lustre-OST0000 complete [ 6735.601605] Lustre: server umount lustre-MDT0000 complete [ 6739.461498] LustreError: 226623:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786750478 with bad export cookie 105567840114737232 [ 6739.469740] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6739.471674] LustreError: 226623:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6740.049661] Lustre: server umount lustre-MDT0001 complete [ 6767.676493] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 19:35:04 (1786750504) [ 6776.363778] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6776.763501] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6776.838736] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6776.847225] Lustre: Skipped 2 previous similar messages [ 6782.268039] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 6792.525575] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6792.844537] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6792.922550] Lustre: 233326:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6798.289537] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 6806.059780] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6812.310192] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6819.326257] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6819.526336] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6819.545120] Lustre: Permanently deactivating lustre-OST0000 [ 6819.566919] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 6825.883153] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 6829.090113] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6834.139296] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6841.291406] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6841.416215] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6848.780108] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 6849.809197] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 6857.241661] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6859.560367] Lustre: Permanently reactivating lustre-OST0000 [ 6859.582575] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 6859.589475] Lustre: Skipped 2 previous similar messages [ 6865.989685] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6865.991654] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 37 seconds. I think it's dead, and I am evicting it. exp 00000000e8367e4f, cur 1786750605 deadline 1786750598 last 1786750568 [ 6865.998124] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6866.025673] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6866.034079] Lustre: Skipped 1 previous similar message [ 6866.101074] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6881.376547] Lustre: server umount lustre-OST0001 complete [ 6891.214155] Lustre: server umount lustre-OST0000 complete [ 6901.925446] Lustre: server umount lustre-MDT0000 complete [ 6906.463770] LustreError: 233319:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786750645 with bad export cookie 105567840114738905 [ 6906.483417] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6906.486676] LustreError: 233319:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6907.071983] Lustre: server umount lustre-MDT0001 complete [ 6919.280355] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 6922.560252] Key type lgssc unregistered [ 6922.885805] LNet: 238214:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6922.898363] LNetError: 238214:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6923.958124] LNet: Removed LNI 192.168.206.151@tcp [ 6925.319789] Key type .llcrypt unregistered [ 6925.324116] Key type ._llcrypt unregistered [ 6950.926828] Key type ._llcrypt registered [ 6950.929789] Key type .llcrypt registered [ 6951.045853] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 6952.026250] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6952.061071] alg: No test for adler32 (adler32-zlib) [ 6953.434524] Lustre: Lustre: Build Version: 2.17.57_1_g59aa375 [ 6953.740895] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 6955.480681] Key type lgssc registered [ 6956.984435] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6967.511394] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6967.530745] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6969.010696] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6974.394826] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 6979.557388] LustreError: 239684:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6979.577473] LustreError: 239684:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6983.970484] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6984.126649] Lustre: Found index 1 for lustre-MDT0001, updating log [ 6984.166336] Lustre: 239672:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6984.179865] Lustre: Permanently deactivating lustre-MDT0001 [ 6984.204547] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 6984.626742] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6989.317362] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 6992.926261] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 6998.648594] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7005.550649] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7015.547158] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7015.930505] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7022.284853] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7029.221258] LustreError: 242145:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7029.240776] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 7030.558789] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7034.341231] LustreError: 242145:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7037.972579] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7038.284703] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7039.367541] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7039.370115] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7039.563700] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7044.686848] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7052.949995] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7055.865773] Lustre: Permanently reactivating lustre-MDT0001 [ 7055.880826] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7055.890613] Lustre: Skipped 2 previous similar messages [ 7060.867996] Lustre: lustre-MDT0001: Client db7275b9-ff84-4edb-a450-1f0a8a70447a (at 192.168.206.51@tcp) reconnecting [ 7062.685870] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7062.702079] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 7062.708268] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 7062.718713] LustreError: 243267:0:(llog_osd.c:2268:llog_osd_get_cat_list()) lustre-MDT0001-osp-MDT0000: error reading CATALOGS: rc = -108 [ 7062.719749] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7062.729875] LustreError: 243267:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -108 [ 7062.729981] LustreError: 243267:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.206.151@tcp: cfg command failed: rc = -108 [ 7062.730025] Lustre: cmd=cf00f 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001-osp-MDT0000.active=1 [ 7062.730025] [ 7062.730954] LustreError: 239675:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 7067.731545] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7069.821142] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7075.597973] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7075.873860] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7080.545914] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7080.784075] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7083.462355] Lustre: Permanently deactivating lustre-MDT0001 [ 7083.499613] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7083.504743] Lustre: Skipped 2 previous similar messages [ 7091.849897] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7102.432257] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7102.443842] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7102.464693] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7106.020117] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7106.032258] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7106.036022] Lustre: Skipped 1 previous similar message [ 7106.050834] Lustre: Skipped 3 previous similar messages [ 7107.776790] Lustre: server umount lustre-MDT0000 complete [ 7111.146627] LustreError: 243742:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7111.167252] LustreError: 243742:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 7111.878585] LustreError: 243231:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786750851 with bad export cookie 9145267677882661822 [ 7111.889267] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7111.894375] LustreError: 243231:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7112.394384] Lustre: server umount lustre-MDT0001 complete [ 7118.556447] Lustre: server umount lustre-OST0000 complete [ 7130.876616] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 19:41:08 (1786750868) [ 7142.779731] Lustre: server umount lustre-OST0001 complete [ 7151.265735] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_hostid [ 7162.427305] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 7202.260413] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7202.470089] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7202.479147] Lustre: Skipped 2 previous similar messages [ 7202.499130] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7202.560855] Lustre: lustre-MDT0000: new disk, initializing [ 7202.632204] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7202.648039] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7206.686931] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7217.280976] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7217.386812] Lustre: 247992:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7217.434709] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7217.437595] Lustre: Skipped 1 previous similar message [ 7217.509672] Lustre: lustre-MDT0001: new disk, initializing [ 7217.588770] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7217.635381] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7217.645675] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7222.080659] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7231.205513] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7236.141937] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7242.000828] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7242.307172] Lustre: lustre-OST0000: new disk, initializing [ 7242.311113] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7242.321472] Lustre: 249959:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7242.415748] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7249.010413] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7251.497969] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7251.513579] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7251.614574] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7259.106414] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7261.769883] LustreError: 247999:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7270.913656] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7271.094938] Lustre: lustre-OST0001: new disk, initializing [ 7271.100563] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7271.114505] Lustre: 251077:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7271.863552] LustreError: 247999:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7271.901980] LustreError: 247998:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7278.888423] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7279.764406] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7279.779756] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7279.958752] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7281.951140] LustreError: 247998:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7288.818783] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7290.437848] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7292.313664] Lustre: lustre-OST0001: Not available for connect from 192.168.206.51@tcp (stopping) [ 7295.467884] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7295.468869] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7295.470867] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7295.485206] Lustre: Skipped 2 previous similar messages [ 7295.503266] LustreError: Skipped 1 previous similar message [ 7298.491337] Lustre: server umount lustre-OST0001 complete [ 7300.585475] LustreError: 249943:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7300.609322] LustreError: 249943:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7310.818675] LustreError: 250424:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7310.852663] LustreError: 250424:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 7315.940789] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7315.947340] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7315.955977] Lustre: Skipped 1 previous similar message [ 7315.961785] Lustre: Skipped 2 previous similar messages [ 7321.760858] Lustre: server umount lustre-OST0000 complete [ 7326.176530] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7326.192866] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7326.193519] Lustre: Skipped 1 previous similar message [ 7326.215848] Lustre: Skipped 4 previous similar messages [ 7332.441483] Lustre: server umount lustre-MDT0000 complete [ 7336.371522] LustreError: 249969:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786751075 with bad export cookie 9145267677882664636 [ 7336.377402] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7336.388851] LustreError: 249969:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7342.665880] Lustre: server umount lustre-MDT0001 complete [ 7352.376872] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 7355.285440] Key type lgssc unregistered [ 7355.576515] LNet: 253116:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7355.589513] LNetError: 253116:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7356.646640] LNet: Removed LNI 192.168.206.151@tcp [ 7357.547207] Key type .llcrypt unregistered [ 7357.553768] Key type ._llcrypt unregistered [ 7396.906947] Key type ._llcrypt registered [ 7396.908558] Key type .llcrypt registered [ 7397.015989] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 7398.378568] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7398.403723] alg: No test for adler32 (adler32-zlib) [ 7399.505116] Lustre: Lustre: Build Version: 2.17.57_1_g59aa375 [ 7399.770560] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 7401.415244] Key type lgssc registered [ 7402.457864] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7410.013486] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7410.057112] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7411.446669] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7411.513206] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7411.705034] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7416.535361] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7424.325103] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7424.432559] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7424.477491] Lustre: 255321:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7424.744442] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7429.344192] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7437.654759] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7443.585118] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7450.057937] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7450.182531] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7450.361795] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7455.887731] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7456.948256] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 7463.436455] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7518.176095] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7518.185580] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7518.202304] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7518.698674] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7518.707754] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7518.715549] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7520.874105] Lustre: server umount lustre-OST0000 complete [ 7526.242159] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7540.348124] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7540.744780] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7546.898474] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7553.860516] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7559.082625] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7559.109559] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7559.113225] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7594.976517] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7594.982317] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7594.987185] LustreError: Skipped 1 previous similar message [ 7594.988215] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7595.032869] Lustre: Skipped 1 previous similar message [ 7599.015179] Lustre: server umount lustre-OST0000 complete [ 7604.193205] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7604.201044] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7604.219973] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7604.228952] Lustre: Skipped 1 previous similar message [ 7609.622821] Lustre: server umount lustre-MDT0000 complete [ 7613.460890] LustreError: 256063:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786751352 with bad export cookie 5370767375901426849 [ 7613.470205] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7613.473948] LustreError: 256063:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7613.929182] Lustre: server umount lustre-MDT0001 complete [ 7625.988087] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 7629.585602] Key type lgssc unregistered [ 7629.912579] LNet: 260745:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7629.918809] LNetError: 260745:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7630.953063] LNet: Removed LNI 192.168.206.151@tcp [ 7632.008616] Key type .llcrypt unregistered [ 7632.011164] Key type ._llcrypt unregistered [ 7657.527854] Key type ._llcrypt registered [ 7657.531758] Key type .llcrypt registered [ 7657.738483] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 7659.300431] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7659.357507] alg: No test for adler32 (adler32-zlib) [ 7660.553637] Lustre: Lustre: Build Version: 2.17.57_1_g59aa375 [ 7660.837468] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 7662.519227] Key type lgssc registered [ 7664.004981] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7671.635342] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7671.656793] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7673.138298] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7677.589614] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7683.554396] LustreError: 262164:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7683.562905] LustreError: 262164:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7686.821048] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7687.333534] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7691.933902] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7701.957390] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7709.870563] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7717.715592] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7718.058486] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7719.106899] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 7725.937490] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7735.914494] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7763.436118] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7763.437822] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7763.456437] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7768.546626] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7768.554713] Lustre: Skipped 2 previous similar messages [ 7768.974051] Lustre: server umount lustre-OST0000 complete [ 7773.667169] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7773.682341] Lustre: Skipped 1 previous similar message [ 7773.694431] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7773.702198] Lustre: Skipped 1 previous similar message [ 7778.788828] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7778.803371] Lustre: Skipped 1 previous similar message [ 7779.214783] Lustre: server umount lustre-MDT0000 complete [ 7783.293236] LustreError: 262144:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786751522 with bad export cookie 9260350555068197293 [ 7783.298520] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7783.311301] LustreError: 262144:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7783.678633] Lustre: server umount lustre-MDT0001 complete [ 7797.149589] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 7800.402424] Key type lgssc unregistered [ 7800.616531] LNet: 266475:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7800.625718] LNetError: 266475:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7800.642777] LNet: Removed LNI 192.168.206.151@tcp [ 7801.288187] Key type .llcrypt unregistered [ 7801.289805] Key type ._llcrypt unregistered [ 7817.981234] Key type ._llcrypt registered [ 7817.982873] Key type .llcrypt registered [ 7818.095225] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 7819.357412] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7819.444273] alg: No test for adler32 (adler32-zlib) [ 7820.539687] Lustre: Lustre: Build Version: 2.17.57_1_g59aa375 [ 7820.789168] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 7822.503183] Key type lgssc registered [ 7823.925981] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7831.560406] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7831.585358] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7832.991494] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7836.812283] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7843.300717] LustreError: 267695:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7843.317724] LustreError: 267695:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7845.232757] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7845.612772] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7849.596565] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7856.782680] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7862.105526] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7867.991341] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7868.241057] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7873.716310] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7878.523641] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 7879.970654] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7893.918966] Lustre: server umount lustre-OST0000 complete [ 7899.108020] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7899.125500] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7899.135326] Lustre: Skipped 1 previous similar message [ 7904.232798] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7904.240499] Lustre: Skipped 1 previous similar message [ 7907.365681] Lustre: server umount lustre-MDT0000 complete [ 7909.348703] LustreError: 270137:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7909.385092] LustreError: 270137:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7910.367349] Lustre: 267009:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786751633/real 1786751633] req@00000000377282df x1873544803805952/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786751649 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7910.402086] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7910.413606] Lustre: Skipped 1 previous similar message [ 7910.424635] LustreError: 267691:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7910.443074] LustreError: 267691:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7911.173032] LustreError: 267678:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786751650 with bad export cookie 11378188622164653175 [ 7911.175686] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7911.183365] LustreError: 267678:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7911.580398] Lustre: server umount lustre-MDT0001 complete [ 7921.728211] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 7924.470982] Key type lgssc unregistered [ 7924.739868] LNet: 271376:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7924.748083] LNetError: 271376:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7924.769704] LNet: Removed LNI 192.168.206.151@tcp [ 7925.669904] Key type .llcrypt unregistered [ 7925.672435] Key type ._llcrypt unregistered [ 7949.045294] Key type ._llcrypt registered [ 7949.048484] Key type .llcrypt registered [ 7949.179667] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 7950.354032] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7950.393527] alg: No test for adler32 (adler32-zlib) [ 7951.504882] Lustre: Lustre: Build Version: 2.17.57_1_g59aa375 [ 7951.742939] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 7953.407169] Key type lgssc registered [ 7954.379330] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7962.179738] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7962.190897] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7963.526981] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7967.595120] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7972.839188] LustreError: 272794:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7972.851740] LustreError: 272794:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7975.537430] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7975.937873] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7980.308690] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7986.717274] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7991.181892] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7997.490316] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7997.856887] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8003.429565] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 8008.123263] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8009.866339] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8033.778761] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8033.780449] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8033.791056] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8033.793281] Lustre: Skipped 1 previous similar message [ 8033.803695] Lustre: Skipped 1 previous similar message [ 8038.514664] Lustre: server umount lustre-OST0000 complete [ 8042.471397] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8042.486791] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8042.506528] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8044.002240] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8044.003832] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8044.027322] Lustre: Skipped 1 previous similar message [ 8048.051434] Lustre: server umount lustre-MDT0000 complete [ 8049.137023] LustreError: 272794:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8049.169469] LustreError: 272794:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8051.590642] LustreError: 272776:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786751790 with bad export cookie 12772101805142719582 [ 8051.623977] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8051.624947] LustreError: 272776:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8052.043589] Lustre: server umount lustre-MDT0001 complete [ 8063.317307] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 8066.052658] Key type lgssc unregistered [ 8066.442579] LNet: 277101:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8066.469632] LNetError: 277101:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8066.501788] LNet: Removed LNI 192.168.206.151@tcp [ 8067.343563] Key type .llcrypt unregistered [ 8067.345946] Key type ._llcrypt unregistered [ 8085.513370] Key type ._llcrypt registered [ 8085.515071] Key type .llcrypt registered [ 8085.617514] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 8086.572726] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8086.633440] alg: No test for adler32 (adler32-zlib) [ 8087.695795] Lustre: Lustre: Build Version: 2.17.57_1_g59aa375 [ 8087.979683] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 8089.659205] Key type lgssc registered [ 8090.899332] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8098.841327] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8098.863853] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8100.255931] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8104.454781] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 8110.560961] LustreError: 278339:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8110.577424] LustreError: 278339:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8113.523133] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8113.989974] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8118.846796] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 8125.990387] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8132.342742] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8139.675405] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8139.956239] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8145.073655] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8145.893688] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 8154.135738] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8169.956110] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8169.988042] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8169.990857] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8175.101218] LustreError: 280186:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8175.127577] LustreError: 280186:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8175.252522] Lustre: server umount lustre-OST0000 complete [ 8180.198262] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8180.217833] Lustre: Skipped 1 previous similar message [ 8180.222198] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8180.224327] Lustre: Skipped 1 previous similar message [ 8185.314984] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8185.322808] Lustre: Skipped 1 previous similar message [ 8185.553163] Lustre: server umount lustre-MDT0000 complete [ 8189.360991] LustreError: 278303:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786751928 with bad export cookie 10608968965068672897 [ 8189.365307] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8189.375662] LustreError: 278303:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8189.957389] Lustre: server umount lustre-MDT0001 complete [ 8200.898758] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 8204.334870] Key type lgssc unregistered [ 8204.661550] LNet: 282042:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8204.671241] LNetError: 282042:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8205.743251] LNet: Removed LNI 192.168.206.151@tcp [ 8206.632427] Key type .llcrypt unregistered [ 8206.634588] Key type ._llcrypt unregistered [ 8220.232233] Key type ._llcrypt registered [ 8220.235390] Key type .llcrypt registered [ 8220.461899] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8239.165222] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_hostid [ 8254.295628] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 8255.387881] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8255.422479] alg: No test for adler32 (adler32-zlib) [ 8256.614553] Lustre: Lustre: Build Version: 2.17.57_1_g59aa375 [ 8256.911992] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 8258.663196] Key type lgssc registered [ 8259.919475] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8294.938698] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8295.008688] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8296.261560] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8296.287756] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8296.355580] Lustre: lustre-MDT0000: new disk, initializing [ 8296.415416] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8296.458215] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8301.154708] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 8312.107786] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8312.209498] Lustre: 285596:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8312.256079] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8312.259826] Lustre: Skipped 1 previous similar message [ 8312.323984] Lustre: lustre-MDT0001: new disk, initializing [ 8312.422041] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8312.478798] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8312.499740] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8316.844248] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 8326.509763] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8332.341649] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8339.801689] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8340.178804] Lustre: lustre-OST0000: new disk, initializing [ 8340.182788] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8340.194723] Lustre: 287576:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8340.344343] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8346.078339] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 8346.189696] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8346.203369] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8346.345191] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8356.643081] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8361.265523] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8361.476781] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8367.187443] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8367.576735] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8372.194598] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8372.206448] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8372.206971] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8372.211795] Lustre: Skipped 1 previous similar message [ 8372.230554] Lustre: Skipped 1 previous similar message [ 8375.757417] Lustre: server umount lustre-OST0000 complete [ 8382.436877] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8382.441541] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8382.455548] Lustre: Skipped 1 previous similar message [ 8385.868439] Lustre: server umount lustre-MDT0000 complete [ 8389.238499] LustreError: 285589:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786752128 with bad export cookie 9393713991098833314 [ 8389.241336] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8389.249285] LustreError: 285589:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8389.555882] Lustre: server umount lustre-MDT0001 complete [ 8398.184827] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 20:02:15 (1786752135) [ 8405.042481] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8405.475188] LustreError: 289990:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8405.500425] LustreError: 289990:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 8405.558322] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8410.020689] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 8410.602466] LustreError: 289991:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8415.721168] LustreError: 289990:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8418.335410] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8418.683316] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8422.758866] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 8430.159180] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8435.001668] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8441.257048] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8441.549355] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8447.497957] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 8453.840479] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8464.355841] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8464.367347] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8464.381147] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8464.393143] Lustre: Skipped 1 previous similar message [ 8465.386166] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8468.897756] Lustre: server umount lustre-OST0000 complete [ 8475.107524] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8475.113712] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8475.126919] Lustre: Skipped 1 previous similar message [ 8475.130818] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8475.140026] Lustre: Skipped 2 previous similar messages [ 8478.721830] Lustre: server umount lustre-MDT0000 complete [ 8480.736278] LustreError: 289990:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8480.749217] LustreError: 289990:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8482.590896] LustreError: 289971:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786752221 with bad export cookie 9393713991098834189 [ 8482.609431] LustreError: 289971:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8482.616867] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8483.046371] Lustre: server umount lustre-MDT0001 complete [ 8493.719957] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 8496.202857] Key type lgssc unregistered [ 8496.428965] LNet: 293528:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8496.437677] LNetError: 293528:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8496.451922] LNet: Removed LNI 192.168.206.151@tcp [ 8497.385336] Key type .llcrypt unregistered [ 8497.387166] Key type ._llcrypt unregistered [ 8522.120348] Key type ._llcrypt registered [ 8522.122406] Key type .llcrypt registered [ 8522.211726] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_hostid [ 8536.300752] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 8537.213196] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8537.296621] alg: No test for adler32 (adler32-zlib) [ 8538.339594] Lustre: Lustre: Build Version: 2.17.57_1_g59aa375 [ 8538.644022] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 8540.351165] Key type lgssc registered [ 8541.425702] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8584.804248] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8584.851598] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8586.059805] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8586.090516] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8586.177695] Lustre: lustre-MDT0000: new disk, initializing [ 8586.279435] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8586.296636] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8590.918611] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 8601.422411] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8601.563270] Lustre: 297357:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8601.634620] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8601.638274] Lustre: Skipped 1 previous similar message [ 8601.755366] Lustre: lustre-MDT0001: new disk, initializing [ 8601.894995] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8601.963550] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8601.975162] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8606.441318] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 8616.486713] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8622.595956] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8629.949765] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8630.286320] Lustre: lustre-OST2710: new disk, initializing [ 8630.294608] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 8630.309400] Lustre: 299327:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8630.428679] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 8636.580965] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 8637.545810] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 8637.559753] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 8637.686045] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 8647.089493] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8655.548063] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8655.758839] Lustre: lustre-OST03e8: new disk, initializing [ 8655.762057] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 8655.770506] Lustre: 300389:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8655.913807] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 8663.444808] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 8664.163039] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 8664.167961] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 8664.269743] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 8670.914132] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8678.725298] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 8679.023199] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8683.599690] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 8683.798139] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8688.734776] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 8688.963432] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8693.875219] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 8694.222158] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8700.403363] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8700.414650] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8700.436191] Lustre: Skipped 3 previous similar messages [ 8700.451793] Lustre: Skipped 3 previous similar messages [ 8705.509465] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8705.517416] Lustre: Skipped 1 previous similar message [ 8705.892898] Lustre: server umount lustre-MDT0000 complete [ 8710.150258] LustreError: 299324:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786752449 with bad export cookie 11756577484399724958 [ 8710.162755] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8710.169144] LustreError: 299324:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8710.588400] Lustre: server umount lustre-MDT0001 complete [ 8724.901415] Lustre: server umount lustre-OST2710 complete [ 8725.989600] Lustre: 295153:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786752449/real 1786752449] req@0000000011effef0 x1873545555678208/t0(0) o400->lustre-MDT0001-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786752465 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8726.012258] Lustre: lustre-MDT0001-lwp-OST03e8: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8729.112991] Lustre: server umount lustre-OST03e8 complete [ 8747.818996] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_hostid [ 8757.619983] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 8791.414856] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8791.740606] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8791.772192] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8791.908665] Lustre: lustre-MDT0000: new disk, initializing [ 8791.989464] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8792.003133] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8797.322308] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 8808.500107] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8808.605743] Lustre: 305638:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8808.673856] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8808.680517] Lustre: Skipped 1 previous similar message [ 8808.781208] Lustre: lustre-MDT0001: new disk, initializing [ 8808.872548] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8808.901315] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8808.921544] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8813.189112] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 8823.459906] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8829.241533] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8837.252592] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8837.557981] Lustre: lustre-OST0000: new disk, initializing [ 8837.561644] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8837.567528] Lustre: 307607:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8837.649848] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8843.349150] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8843.362459] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8843.480920] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8844.744082] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 8854.281251] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8859.086781] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8859.296692] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8862.822692] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8863.002846] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8869.351271] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8869.351691] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8869.370062] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8874.467893] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8874.478206] Lustre: Skipped 1 previous similar message [ 8879.071243] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8879.202194] Lustre: server umount lustre-OST0000 complete [ 8884.707504] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8884.724735] Lustre: Skipped 1 previous similar message [ 8884.729167] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8884.732673] Lustre: Skipped 2 previous similar messages [ 8888.531469] Lustre: server umount lustre-MDT0000 complete [ 8892.479731] LustreError: 307605:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786752631 with bad export cookie 11756577484399726645 [ 8892.484690] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8892.494277] LustreError: 307605:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8892.855966] Lustre: server umount lustre-MDT0001 complete [ 8902.893715] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 20:10:40 (1786752640) [ 8905.039430] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 8907.479388] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 20:10:44 (1786752644) [ 8912.036506] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 8939.338079] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8939.510172] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8939.547783] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8939.712585] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8944.649045] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 8953.601180] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8953.736731] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8953.787151] Lustre: 311473:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8959.063592] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 8980.773053] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8981.012141] Lustre: Denying initial registration attempt from nid 192.168.206.151@tcp, specified as failover [ 8981.023561] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 8981.030382] LustreError: 312925:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -99 [ 8981.035837] LustreError: 312925:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 8981.041278] LustreError: 312925:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 8981.131458] Lustre: server umount lustre-OST0000 complete [ 8981.134737] LustreError: 312925:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 8990.177270] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8990.182596] LustreError: Skipped 1 previous similar message [ 8990.184526] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8990.197340] Lustre: Skipped 1 previous similar message [ 8990.201255] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8993.927959] Lustre: server umount lustre-MDT0000 complete [ 8995.808023] LustreError: 312930:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8995.819209] LustreError: 312930:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 8997.763346] LustreError: 311465:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786752737 with bad export cookie 11756577484399727534 [ 8997.776638] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8998.134809] Lustre: server umount lustre-MDT0001 complete [ 9008.836477] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 9012.028861] Key type lgssc unregistered [ 9012.270833] LNet: 314051:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9012.282868] LNetError: 314051:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9012.303342] LNet: Removed LNI 192.168.206.151@tcp [ 9013.116877] Key type .llcrypt unregistered [ 9013.119722] Key type ._llcrypt unregistered [ 9024.767356] Key type ._llcrypt registered [ 9024.769340] Key type .llcrypt registered [ 9024.878575] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 9025.947328] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9025.974958] alg: No test for adler32 (adler32-zlib) [ 9027.089169] Lustre: Lustre: Build Version: 2.17.57_1_g59aa375 [ 9027.435768] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 9029.117252] Key type lgssc registered [ 9030.391252] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9079.019684] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 9087.720456] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9087.746660] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9089.018029] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9089.070621] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9089.285496] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9094.774434] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 9104.357407] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9104.425824] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9104.462314] Lustre: 316813:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9104.778095] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9109.355865] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 9117.518703] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9123.632039] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9143.277835] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9143.451898] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9143.630425] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9149.552634] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 9156.904296] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9159.143456] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9159.143830] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9159.145529] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9159.151429] LustreError: Skipped 1 previous similar message [ 9159.175681] Lustre: Skipped 1 previous similar message [ 9160.899342] Lustre: server umount lustre-OST0000 complete [ 9164.260939] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9164.278040] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9164.286049] Lustre: Skipped 1 previous similar message [ 9164.294857] Lustre: Skipped 1 previous similar message [ 9169.376612] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9169.383897] Lustre: Skipped 2 previous similar messages [ 9170.383333] Lustre: server umount lustre-MDT0000 complete [ 9173.668206] LustreError: 316806:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786752912 with bad export cookie 10941934196980472615 [ 9173.671597] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9173.678888] LustreError: 316806:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9173.942768] Lustre: server umount lustre-MDT0001 complete [ 9184.375776] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 9187.178251] Key type lgssc unregistered [ 9187.523113] LNet: 320388:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9187.530969] LNetError: 320388:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9187.543991] LNet: Removed LNI 192.168.206.151@tcp [ 9188.368425] Key type .llcrypt unregistered [ 9188.371151] Key type ._llcrypt unregistered [ 9213.688314] Key type ._llcrypt registered [ 9213.691209] Key type .llcrypt registered [ 9213.835293] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 9215.209878] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9215.261099] alg: No test for adler32 (adler32-zlib) [ 9216.366769] Lustre: Lustre: Build Version: 2.17.57_1_g59aa375 [ 9216.716214] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 9218.447242] Key type lgssc registered [ 9220.116766] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9228.784153] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9228.805718] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9230.201989] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9234.689536] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 9240.552262] LustreError: 321806:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9240.570224] LustreError: 321806:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9243.388864] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9243.911152] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9248.684833] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 9255.990924] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9261.559532] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9270.441101] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9270.824892] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9277.105533] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 9284.264819] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9299.426545] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9299.429452] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9299.435036] Lustre: Skipped 1 previous similar message [ 9299.445160] Lustre: Skipped 2 previous similar messages [ 9301.373974] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.51@tcp (stopping) [ 9303.985204] Lustre: server umount lustre-MDT0000 complete [ 9304.547448] LustreError: 321807:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9304.563507] LustreError: 321807:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9306.473658] LustreError: 323625:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9307.831050] LustreError: 321786:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786753047 with bad export cookie 1422049563041897847 [ 9307.842819] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9307.848116] LustreError: 321786:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9309.664385] LustreError: 321801:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9309.664620] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9309.664636] Lustre: Skipped 1 previous similar message [ 9309.666208] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9312.675076] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9312.687185] Lustre: Skipped 1 previous similar message [ 9314.247720] Lustre: server umount lustre-MDT0001 complete [ 9321.183295] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9331.250987] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9333.216924] LustreError: 321115:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000003942c038 x1873546266582144/t0(0) o250->MGC192.168.206.151@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9333.598984] LustreError: 323641:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9333.621909] LustreError: 323641:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9333.674376] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9333.736671] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 9338.204854] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 9342.883690] LustreError: 325424:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9342.895542] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9342.917513] LustreError: 325424:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9347.718483] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9348.144348] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9353.000547] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 9353.202760] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9359.875685] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9361.687536] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9367.447709] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9369.147891] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9374.694280] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9374.698285] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9374.706799] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9378.788188] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9378.796303] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9380.538166] Lustre: server umount lustre-OST0000 complete [ 9389.024693] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9389.025982] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9389.041533] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9389.054801] Lustre: Skipped 3 previous similar messages [ 9390.361420] Lustre: server umount lustre-MDT0000 complete [ 9393.813882] LustreError: 321786:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786753133 with bad export cookie 1422049563041905435 [ 9393.820199] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9393.828869] LustreError: 321786:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9394.073704] Lustre: server umount lustre-MDT0001 complete [ 9403.978512] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 9406.943948] Key type lgssc unregistered [ 9407.213139] LNet: 328096:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9407.227921] LNetError: 328096:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9407.249190] LNet: Removed LNI 192.168.206.151@tcp [ 9408.147301] Key type .llcrypt unregistered [ 9408.152159] Key type ._llcrypt unregistered [ 9431.936126] Key type ._llcrypt registered [ 9431.938560] Key type .llcrypt registered [ 9432.063273] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 9432.879560] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9432.918351] alg: No test for adler32 (adler32-zlib) [ 9434.092324] Lustre: Lustre: Build Version: 2.17.57_1_g59aa375 [ 9434.349533] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 9436.072799] Key type lgssc registered [ 9437.060644] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9443.889215] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9443.920238] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9445.349173] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9449.896032] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 9455.585877] LustreError: 329513:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9455.599406] LustreError: 329513:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9458.349975] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9458.668986] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9462.745382] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 9469.811588] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9475.029311] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9478.088642] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9478.099100] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9478.109592] Lustre: Skipped 1 previous similar message [ 9483.233456] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9483.243089] Lustre: Skipped 1 previous similar message [ 9484.626347] Lustre: server umount lustre-MDT0000 complete [ 9488.215463] LustreError: 329492:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786753227 with bad export cookie 11639355065329341101 [ 9488.230605] LustreError: 329492:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9488.231974] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9488.661859] Lustre: server umount lustre-MDT0001 complete [ 9502.885083] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9503.123939] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9503.168965] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9503.324290] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9507.152708] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 9515.182559] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9515.259677] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9515.290249] Lustre: 332101:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9515.620377] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9519.618318] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 9527.214881] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9533.154960] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9543.036229] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9543.293754] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 9543.305712] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 9543.318885] LustreError: 333984:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 9543.334693] LustreError: 333984:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 9543.343045] LustreError: 333984:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9543.408218] Lustre: server umount lustre-OST0000 complete [ 9543.413409] LustreError: 333984:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 9554.433116] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9554.654216] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9554.872100] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9560.763741] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 9562.808628] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 9573.254282] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9581.647829] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9581.792240] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 9581.876522] Lustre: lustre-OST0001: new disk, initializing [ 9581.885284] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9581.892683] Lustre: 335525:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9582.039349] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9587.735986] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 9591.429647] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9591.447818] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9591.579950] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9595.501679] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9602.027801] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9602.028956] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9602.044792] LustreError: Skipped 1 previous similar message [ 9602.045528] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9602.056704] Lustre: Skipped 2 previous similar messages [ 9607.141500] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9607.146814] Lustre: Skipped 2 previous similar messages [ 9612.277117] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9612.282607] Lustre: Skipped 1 previous similar message [ 9613.279557] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9613.359609] Lustre: server umount lustre-OST0001 complete [ 9616.869796] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9616.876308] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9617.381716] LustreError: 334448:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9617.400367] LustreError: 334448:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9622.504055] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9622.507779] LustreError: 334449:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9622.522223] Lustre: Skipped 4 previous similar messages [ 9627.626641] LustreError: 334449:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9627.653838] LustreError: 334449:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9631.199149] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9631.318609] Lustre: server umount lustre-OST0000 complete [ 9637.856467] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9637.882045] Lustre: Skipped 2 previous similar messages [ 9641.277530] Lustre: server umount lustre-MDT0000 complete [ 9642.976654] LustreError: 332108:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9643.003433] LustreError: 332108:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9644.805529] LustreError: 332092:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786753384 with bad export cookie 11639355065329341654 [ 9644.808290] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9644.821527] LustreError: 332092:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9645.107923] Lustre: server umount lustre-MDT0001 complete [ 9655.745792] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 9658.222852] Key type lgssc unregistered [ 9658.506571] LNet: 337489:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9658.524072] LNetError: 337489:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9658.548295] LNet: Removed LNI 192.168.206.151@tcp [ 9659.322272] Key type .llcrypt unregistered [ 9659.324343] Key type ._llcrypt unregistered [ 9734.296358] Key type ._llcrypt registered [ 9734.299752] Key type .llcrypt registered [ 9734.390900] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_hostid [ 9750.503427] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 9751.639098] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9751.737530] alg: No test for adler32 (adler32-zlib) [ 9752.842570] Lustre: Lustre: Build Version: 2.17.57_1_g59aa375 [ 9753.062540] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 9754.735166] Key type lgssc registered [ 9755.682689] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9793.331668] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9793.351449] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9794.644985] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9794.680917] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9794.811949] Lustre: lustre-MDT0000: new disk, initializing [ 9794.920186] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9794.948983] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9799.262781] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 9809.906984] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9810.054716] Lustre: 342423:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9810.134198] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9810.138728] Lustre: Skipped 1 previous similar message [ 9810.240495] Lustre: lustre-MDT0001: new disk, initializing [ 9810.344842] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9810.395244] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9810.417464] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9815.100171] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 9825.348498] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9830.792232] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9837.336173] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9837.604626] Lustre: lustre-OST0000: new disk, initializing [ 9837.609195] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9837.615217] Lustre: 344394:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9837.713053] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9844.647468] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 9845.875979] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9845.886643] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9846.064043] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9855.314738] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9860.057538] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9860.286623] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9865.216667] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9865.437130] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9871.860226] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9871.863289] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9871.875158] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9871.883042] LustreError: Skipped 1 previous similar message [ 9871.894323] Lustre: Skipped 1 previous similar message [ 9873.834880] Lustre: server umount lustre-OST0000 complete [ 9882.080195] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9882.082384] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9882.088153] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9882.107316] Lustre: Skipped 2 previous similar messages [ 9883.913819] Lustre: server umount lustre-MDT0000 complete [ 9887.208514] LustreError: 344298:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9887.237938] LustreError: 344298:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9887.589156] LustreError: 343219:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786753626 with bad export cookie 7024632179097166724 [ 9887.599628] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9887.604809] LustreError: 343219:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9887.937276] Lustre: server umount lustre-MDT0001 complete [ 9896.381838] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 20:27:14 (1786753634) [ 9903.828779] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 20:27:21 (1786753641) [ 9913.002847] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9913.461436] LustreError: 347149:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9913.529691] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9917.659604] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 9918.946872] LustreError: 347150:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9923.043699] LustreError: 347149:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9926.684556] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9926.958901] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9931.594203] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 9939.183708] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9944.430307] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9949.961983] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9950.257333] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9955.734147] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 9961.803111] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9969.177942] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 9970.937438] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 9972.437452] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 9973.999560] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 9975.509869] Lustre: Failing over lustre-MDT0000 [ 9977.734700] Lustre: server umount lustre-MDT0000 complete [ 9978.342361] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9978.347593] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9978.366283] LustreError: 347877:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9989.091124] LustreError: 347150:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9989.110366] LustreError: 347150:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 9995.008686] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9995.138334] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9995.424579] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9995.478379] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9999.049375] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [10000.233599] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10000.887886] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10000.921138] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [10000.965114] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10007.064179] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10008.746238] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10012.225402] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [10014.510131] Lustre: Failing over lustre-MDT0000 [10014.920343] Lustre: server umount lustre-MDT0000 complete [10015.630799] LustreError: 347146:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10015.646248] LustreError: 347146:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [10016.224697] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10016.232786] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10016.254101] Lustre: Skipped 4 previous similar messages [10032.614142] Lustre: 340647:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786753755/real 1786753755] req@0000000062cb5918 x1873546829715712/t0(0) o400->MGC192.168.206.151@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786753771 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10032.636123] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10034.696215] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10041.832097] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x617c81bfb0dd6e27 [10041.844588] Lustre: MGC192.168.206.151@tcp: Connection restored to 0@lo (at 0@lo) [10041.850561] Lustre: Skipped 2 previous similar messages [10042.258207] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10042.341191] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10046.311293] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10047.477635] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10047.542624] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [10047.588811] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [10047.802128] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [10058.416781] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10060.721485] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10067.938782] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10067.953881] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10067.965701] Lustre: Skipped 1 previous similar message [10067.975583] Lustre: Skipped 2 previous similar messages [10073.061254] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10073.072650] Lustre: Skipped 2 previous similar messages [10074.118413] Lustre: server umount lustre-MDT0000 complete [10077.822423] LustreError: 347129:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786753817 with bad export cookie 7024632179097169447 [10077.829492] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10077.841711] LustreError: 347129:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10078.123868] Lustre: server umount lustre-MDT0001 complete [10092.499563] Lustre: server umount lustre-OST0000 complete [10107.199697] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10107.556840] LustreError: 353498:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10107.575631] LustreError: 353498:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 26 previous similar messages [10107.618765] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10112.516607] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [10122.539217] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10127.997473] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [10135.911199] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10142.413506] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10150.399784] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10151.789848] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [10157.759949] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [10165.907629] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10177.504359] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10177.513710] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10177.526854] Lustre: Skipped 1 previous similar message [10177.537316] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10182.603570] Lustre: server umount lustre-OST0000 complete [10189.802894] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10189.811082] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10189.812531] LustreError: Skipped 1 previous similar message [10189.824754] Lustre: Skipped 3 previous similar messages [10192.728834] Lustre: server umount lustre-MDT0000 complete [10194.912658] LustreError: 353499:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10194.935283] LustreError: 353499:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [10195.972667] LustreError: 353479:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786753935 with bad export cookie 7024632179097170189 [10195.982664] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10195.985149] LustreError: 353479:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10196.318570] Lustre: server umount lustre-MDT0001 complete [10206.960174] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [10209.562509] Key type lgssc unregistered [10209.833147] LNet: 357036:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10209.845949] LNetError: 357036:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10209.869131] LNet: Removed LNI 192.168.206.151@tcp [10210.664441] Key type .llcrypt unregistered [10210.667269] Key type ._llcrypt unregistered [10238.270877] Key type ._llcrypt registered [10238.275427] Key type .llcrypt registered [10238.407101] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [10239.615273] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10239.642193] alg: No test for adler32 (adler32-zlib) [10240.659876] Lustre: Lustre: Build Version: 2.17.57_1_g59aa375 [10240.898198] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [10242.567357] Key type lgssc registered [10243.797548] Lustre: Echo OBD driver; http://www.lustre.org/ [10251.410656] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10251.438862] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10252.847471] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10257.933174] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [10263.010048] LustreError: 358552:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10263.022250] LustreError: 358552:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10267.032943] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10267.473993] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10272.577460] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [10280.730389] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10286.052938] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10292.725086] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10293.021482] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10298.189055] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [10299.285876] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [10308.897615] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10323.936462] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10323.944570] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10323.963257] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10326.096838] Lustre: server umount lustre-MDT0000 complete [10328.034627] LustreError: 358552:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10328.039826] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10328.084917] Lustre: Skipped 1 previous similar message [10330.171406] LustreError: 358534:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786754069 with bad export cookie 12205672186264310170 [10330.181343] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10330.186397] LustreError: 358534:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10330.585250] Lustre: server umount lustre-MDT0001 complete [10345.593426] Lustre: server umount lustre-OST0000 complete [10359.007796] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10359.450840] LustreError: 362284:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10359.470997] LustreError: 362284:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10359.534881] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10363.994475] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [10364.902060] LustreError: 362285:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10369.001459] LustreError: 362284:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10373.803791] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10374.118680] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [10374.301714] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10379.701194] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [10388.368861] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10394.863379] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10401.692874] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10401.945100] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10404.024543] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [10408.876619] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [10416.772399] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10429.931456] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10429.941170] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10429.953064] Lustre: Skipped 1 previous similar message [10429.959340] Lustre: Skipped 2 previous similar messages [10435.045286] LustreError: 362284:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10435.071336] LustreError: 362284:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10435.209976] Lustre: server umount lustre-MDT0000 complete [10438.979898] LustreError: 362266:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786754178 with bad export cookie 12205672186264311283 [10438.981971] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10438.986777] LustreError: 362266:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10439.310686] Lustre: server umount lustre-MDT0001 complete [10454.056732] Lustre: server umount lustre-OST0000 complete [10470.818907] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10471.268668] LustreError: 366196:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10471.348502] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10476.158327] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [10485.116886] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10489.414649] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [10497.109488] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10502.913139] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10509.506425] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10509.780741] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10509.791000] Lustre: Skipped 1 previous similar message [10514.939839] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [10515.965851] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [10521.811724] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10528.641039] Lustre: server umount lustre-OST0000 complete [10536.416530] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10536.422143] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10536.432249] Lustre: Skipped 2 previous similar messages [10541.536483] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10541.539478] Lustre: Skipped 1 previous similar message [10542.648902] Lustre: server umount lustre-MDT0000 complete [10546.076541] LustreError: 368041:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786754285 with bad export cookie 12205672186264312305 [10546.079203] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10546.089095] LustreError: 368041:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10556.515542] Lustre: server umount lustre-MDT0001 complete [10568.988354] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [10572.772727] Key type lgssc unregistered [10573.138539] LNet: 369736:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10573.150697] LNetError: 369736:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10574.203207] LNet: Removed LNI 192.168.206.151@tcp [10575.427321] Key type .llcrypt unregistered [10575.433877] Key type ._llcrypt unregistered [10597.878837] Key type ._llcrypt registered [10597.881741] Key type .llcrypt registered [10598.039577] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [10599.070325] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10599.094471] alg: No test for adler32 (adler32-zlib) [10600.247987] Lustre: Lustre: Build Version: 2.17.57_1_g59aa375 [10600.580533] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [10602.279355] Key type lgssc registered [10603.349652] Lustre: Echo OBD driver; http://www.lustre.org/ [10612.081722] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10612.117073] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10612.124420] LustreError: 371172:0:(osd_handler.c:8553:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [10612.150377] LustreError: 371172:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [10612.154541] LustreError: 371172:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [10612.164830] LustreError: 371172:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [10612.174414] LustreError: 371172:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [10620.444633] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [10620.455479] LustreError: 371569:0:(osd_handler.c:8553:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [10620.481282] LustreError: 371569:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [10620.501413] LustreError: 371569:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [10620.519838] LustreError: 371569:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [10620.532067] LustreError: 371569:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [10641.244224] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [10644.356668] Key type lgssc unregistered [10644.675720] LNet: 372577:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10644.683421] LNetError: 372577:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10644.701629] LNet: Removed LNI 192.168.206.151@tcp [10645.495542] Key type .llcrypt unregistered [10645.502930] Key type ._llcrypt unregistered [10664.538221] Key type ._llcrypt registered [10664.542740] Key type .llcrypt registered [10664.622359] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_hostid [10680.936947] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [10682.233872] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10682.291513] alg: No test for adler32 (adler32-zlib) [10683.310266] Lustre: Lustre: Build Version: 2.17.57_1_g59aa375 [10683.616533] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [10685.287181] Key type lgssc registered [10686.564706] Lustre: Echo OBD driver; http://www.lustre.org/ [10728.360803] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10728.393243] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10729.797695] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10729.878092] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10730.073319] Lustre: lustre-MDT0000: new disk, initializing [10730.191240] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10730.222287] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10734.561845] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [10745.238275] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10745.341491] Lustre: 375736:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10745.411758] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10745.419402] Lustre: Skipped 1 previous similar message [10745.495517] Lustre: lustre-MDT0001: new disk, initializing [10745.573317] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10745.616628] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10745.631752] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10749.254517] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [10757.846363] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10763.913578] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10770.465588] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10770.801896] Lustre: lustre-OST0000: new disk, initializing [10770.807401] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10770.815375] Lustre: 377704:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10770.893500] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10776.088620] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [10777.148279] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10777.155720] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10777.270270] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10784.418100] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10788.937241] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10789.149225] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10793.626868] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10793.802823] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10798.050757] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10798.059098] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10798.066622] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10798.068953] Lustre: Skipped 1 previous similar message [10798.079282] Lustre: Skipped 1 previous similar message [10803.178562] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10803.187465] Lustre: Skipped 1 previous similar message [10808.300265] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10808.314876] Lustre: Skipped 1 previous similar message [10810.335603] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10810.637419] Lustre: server umount lustre-OST0000 complete [10817.505660] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10817.515870] LustreError: Skipped 1 previous similar message [10817.522091] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10817.533787] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10818.533969] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10820.369974] Lustre: server umount lustre-MDT0000 complete [10823.455827] LustreError: 375729:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786754562 with bad export cookie 1543406879327769986 [10823.459087] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10823.463502] LustreError: 375729:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10823.774296] Lustre: server umount lustre-MDT0001 complete [10830.970604] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [10832.255655] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 20:42:50 (1786754570) [10837.985827] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10838.372785] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10842.243900] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [10843.618498] LustreError: 380164:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10843.635848] LustreError: 380164:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10848.751700] LustreError: 380165:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10851.381959] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10851.663763] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10856.636858] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [10864.932064] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10871.056555] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10878.452454] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10878.826943] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10884.863582] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [10891.753987] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10898.738592] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10898.917364] Lustre: lustre-OST0001: new disk, initializing [10898.926939] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10898.941325] Lustre: 382999:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10899.007056] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10904.973066] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [10907.287934] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10907.306762] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10907.408311] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10914.884852] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10921.865464] Lustre: lustre-OST0001: Not available for connect from 192.168.206.51@tcp (stopping) [10921.873555] Lustre: Skipped 2 previous similar messages [10922.978047] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10923.004540] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10924.570132] Lustre: server umount lustre-OST0001 complete [10926.964912] LustreError: 381985:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.206.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10933.226978] LustreError: 383467:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10933.262074] LustreError: 383467:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10938.343482] LustreError: 381987:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10938.378725] LustreError: 381987:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10942.432944] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10942.446146] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10942.473179] Lustre: Skipped 1 previous similar message [10942.489475] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10942.505087] Lustre: Skipped 2 previous similar messages [10948.652498] Lustre: server umount lustre-OST0000 complete [10953.697403] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10953.708063] Lustre: Skipped 1 previous similar message [10958.817938] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10958.827917] Lustre: Skipped 5 previous similar messages [10959.693375] Lustre: server umount lustre-MDT0000 complete [10963.664705] LustreError: 380146:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786754702 with bad export cookie 1543406879327770875 [10963.669820] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10963.675727] LustreError: 380146:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10970.073536] Lustre: server umount lustre-MDT0001 complete [10981.420592] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [10984.297714] Key type lgssc unregistered [10984.612551] LNet: 384989:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10984.618226] LNetError: 384989:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10984.656623] LNet: Removed LNI 192.168.206.151@tcp [10985.479519] Key type .llcrypt unregistered [10985.483072] Key type ._llcrypt unregistered [11035.998353] Key type ._llcrypt registered [11036.001424] Key type .llcrypt registered [11036.106941] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [11037.399394] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11037.472200] alg: No test for adler32 (adler32-zlib) [11038.556434] Lustre: Lustre: Build Version: 2.17.57_1_g59aa375 [11038.862686] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [11040.599180] Key type lgssc registered [11041.648203] Lustre: Echo OBD driver; http://www.lustre.org/ [11048.916823] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11048.940527] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11050.168039] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [11050.205423] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11050.368553] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11053.860489] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [11061.575780] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11061.648429] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11061.674535] Lustre: 387784:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11061.906334] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11066.207605] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [11073.049746] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11078.693858] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11082.720246] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11082.730277] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11082.747177] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11085.288555] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11085.290485] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11085.325128] Lustre: Skipped 1 previous similar message [11086.916224] Lustre: server umount lustre-MDT0000 complete [11091.052765] LustreError: 387777:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786754830 with bad export cookie 11403730294216073657 [11091.072136] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11091.085658] LustreError: 387777:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11091.426963] Lustre: server umount lustre-MDT0001 complete [11101.267332] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 20:47:18 (1786754838) [11108.607109] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11109.126119] LustreError: 390293:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11109.150445] LustreError: 390293:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11109.188677] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11113.877953] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [11114.475196] LustreError: 390294:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11119.589988] LustreError: 390293:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11123.415648] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11124.007312] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11129.557884] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [11138.563282] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11144.992284] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11151.773093] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11151.893993] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [11152.066377] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11158.049227] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [11166.040570] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11172.800519] Lustre: Permanently deactivating lustre-OST0000 [11172.817834] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [11178.529881] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11179.687018] LustreError: 393233:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [11179.705889] LustreError: 393233:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11179.711549] LustreError: 393233:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11182.129889] LustreError: 393282:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11182.143593] LustreError: 393282:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11190.128908] Lustre: server umount lustre-OST0000 complete [11192.083599] LustreError: 393531:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11192.089824] LustreError: 393531:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11195.872666] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11195.879484] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11195.899839] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11202.018278] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11202.023747] Lustre: Skipped 3 previous similar messages [11203.125827] Lustre: server umount lustre-MDT0000 complete [11206.664536] LustreError: 392985:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786754945 with bad export cookie 11403730294216074238 [11206.670513] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11206.679501] LustreError: 392985:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11216.062084] Lustre: server umount lustre-MDT0001 complete [11225.010132] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11229.470439] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [11232.295948] LustreError: 394691:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [11239.098437] Lustre: 395029:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.206.151@tcp [11241.491213] Lustre: server umount MGS complete [11254.339167] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11254.874579] LustreError: 395754:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11254.890534] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11254.894291] Lustre: Skipped 1 previous similar message [11254.933787] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11259.510248] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [11260.391463] LustreError: 395755:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11268.895027] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11269.309337] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11274.555728] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [11283.399417] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11289.121309] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11291.329647] Lustre: Permanently reactivating lustre-OST0000 [11291.348608] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [11291.353833] Lustre: Skipped 2 previous similar messages [11320.289989] LustreError: 397583:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [11327.907096] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11328.346809] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11328.356539] Lustre: Skipped 1 previous similar message [11335.422076] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [11343.892984] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11351.010917] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11351.018112] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11351.040926] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11351.046135] Lustre: Skipped 2 previous similar messages [11351.065120] Lustre: Skipped 1 previous similar message [11356.946840] Lustre: server umount lustre-OST0000 complete [11361.248669] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11361.277870] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11361.301899] Lustre: Skipped 3 previous similar messages [11367.700242] Lustre: server umount lustre-MDT0000 complete [11371.491280] LustreError: 395755:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11371.521811] LustreError: 395755:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11371.548568] LustreError: 395736:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786755110 with bad export cookie 11403730294216075288 [11371.554786] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11371.565116] LustreError: 395736:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11372.338042] Lustre: server umount lustre-MDT0001 complete [11383.909793] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [11386.749776] Key type lgssc unregistered [11387.091308] LNet: 400173:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11387.098978] LNetError: 400173:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11387.122518] LNet: Removed LNI 192.168.206.151@tcp [11387.900253] Key type .llcrypt unregistered [11387.904068] Key type ._llcrypt unregistered [11407.014573] Key type ._llcrypt registered [11407.017021] Key type .llcrypt registered [11407.138278] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_hostid [11423.460683] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [11424.753087] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11424.777675] alg: No test for adler32 (adler32-zlib) [11425.781908] Lustre: Lustre: Build Version: 2.17.57_1_g59aa375 [11426.075978] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [11427.807183] Key type lgssc registered [11428.943928] Lustre: Echo OBD driver; http://www.lustre.org/ [11466.270059] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11466.284586] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11467.547308] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11467.582570] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11467.675541] Lustre: lustre-MDT0000: new disk, initializing [11467.793053] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11467.818457] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11471.996335] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [11482.953676] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11483.119313] Lustre: 403330:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11483.199566] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11483.208939] Lustre: Skipped 1 previous similar message [11483.319223] Lustre: lustre-MDT0001: new disk, initializing [11483.428946] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11483.484980] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11483.490529] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11488.386071] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [11497.566790] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11503.086245] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11510.517705] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11510.876789] Lustre: lustre-OST0000: new disk, initializing [11510.885332] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11510.898698] Lustre: 405299:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11511.008398] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11518.576387] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [11518.606607] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11518.640318] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11518.721726] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11528.199931] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11532.737966] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11532.931580] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11537.654420] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11537.941451] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11544.549501] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11544.553483] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11544.561849] LustreError: Skipped 1 previous similar message [11544.566736] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11544.574123] Lustre: Skipped 1 previous similar message [11544.598726] Lustre: Skipped 1 previous similar message [11549.673398] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11549.679890] Lustre: Skipped 1 previous similar message [11554.271132] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11554.557101] Lustre: server umount lustre-OST0000 complete [11559.905565] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11559.918197] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11559.921888] Lustre: Skipped 1 previous similar message [11564.224691] Lustre: server umount lustre-MDT0000 complete [11567.261049] LustreError: 403324:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786755306 with bad export cookie 5869463885353948923 [11567.272626] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11567.277146] LustreError: 403324:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11567.592349] Lustre: server umount lustre-MDT0001 complete [11575.294624] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [11576.824863] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 20:55:14 (1786755314) [11582.749593] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11583.096341] LustreError: 407760:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11583.118017] LustreError: 407760:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11583.164679] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11586.996634] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [11588.576945] LustreError: 407759:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11593.699437] LustreError: 407760:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11594.924294] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11595.162630] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11599.218811] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [11605.863392] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11611.413605] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11619.587033] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11619.827868] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11627.356854] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [11636.151493] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11638.497404] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [11641.268882] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [11648.481861] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11648.510196] Lustre: Skipped 1 previous similar message [11648.537636] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11652.214299] Lustre: server umount lustre-OST0000 complete [11658.728197] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11658.754476] Lustre: Skipped 1 previous similar message [11658.762518] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11658.779145] Lustre: Skipped 2 previous similar messages [11663.668225] Lustre: server umount lustre-MDT0000 complete [11663.843940] LustreError: 407760:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11667.937360] LustreError: 408810:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11667.968145] LustreError: 408810:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11668.529852] LustreError: 407740:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786755407 with bad export cookie 5869463885353949805 [11668.532573] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11668.540928] LustreError: 407740:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11668.998888] Lustre: server umount lustre-MDT0001 complete [11679.405978] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [11682.609509] Key type lgssc unregistered [11683.029024] LNet: 411487:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11683.039642] LNetError: 411487:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11683.062115] LNet: Removed LNI 192.168.206.151@tcp [11684.071461] Key type .llcrypt unregistered [11684.074123] Key type ._llcrypt unregistered [11708.404923] Key type ._llcrypt registered [11708.407192] Key type .llcrypt registered [11708.503916] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [11708.809412] Key type .llcrypt unregistered [11708.811616] Key type ._llcrypt unregistered [11728.115326] Key type ._llcrypt registered [11728.119406] Key type .llcrypt registered [11728.279802] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [11729.649985] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11729.713651] alg: No test for adler32 (adler32-zlib) [11730.840820] Lustre: Lustre: Build Version: 2.17.57_1_g59aa375 [11731.119518] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [11732.815557] Key type lgssc registered [11733.957707] Lustre: Echo OBD driver; http://www.lustre.org/ [11742.390822] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11742.416448] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11744.013403] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11749.820241] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [11754.466229] LustreError: 413711:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11754.484509] LustreError: 413711:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11759.587425] LustreError: 413712:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11760.232577] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11760.768366] LustreError: 413711:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11760.817792] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11766.252520] LustreError: 413712:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11766.264363] LustreError: 413712:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11768.584977] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [11769.376833] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [11774.434941] LustreError: 413708:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11774.460025] LustreError: 413708:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11777.436872] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11784.318766] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11784.703937] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11790.080287] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [11800.042818] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11800.043299] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11800.058437] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11800.067146] Lustre: Skipped 1 previous similar message [11800.074254] LustreError: Skipped 1 previous similar message [11805.161502] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11805.176437] Lustre: Skipped 1 previous similar message [11805.872834] Lustre: server umount lustre-OST0000 complete [11810.292556] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11810.292914] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11810.317602] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11810.328886] Lustre: Skipped 1 previous similar message [11810.338842] Lustre: Skipped 1 previous similar message [11812.278277] Lustre: server umount lustre-MDT0000 complete [11815.395150] LustreError: 413711:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11815.414607] LustreError: 413711:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11816.408455] LustreError: 413693:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786755555 with bad export cookie 18062442066633703798 [11816.419152] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11816.421099] LustreError: 413693:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11816.796963] Lustre: server umount lustre-MDT0001 complete [11828.339516] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [11831.408698] Key type lgssc unregistered [11831.652406] LNet: 416860:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11831.663687] LNetError: 416860:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11831.677055] LNet: Removed LNI 192.168.206.151@tcp [11832.656931] Key type .llcrypt unregistered [11832.662171] Key type ._llcrypt unregistered [11861.245126] Key type ._llcrypt registered [11861.246609] Key type .llcrypt registered [11861.332936] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [11862.292737] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11862.328296] alg: No test for adler32 (adler32-zlib) [11863.600112] Lustre: Lustre: Build Version: 2.17.57_1_g59aa375 [11863.911933] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [11865.639304] Key type lgssc registered [11867.246868] Lustre: Echo OBD driver; http://www.lustre.org/ [11876.527740] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11876.566931] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11892.255617] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [11897.375318] LustreError: 418249:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.151@tcp: failed processing log, type 4: rc = -110 [11924.191532] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11931.473976] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [11940.902580] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11949.547269] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11950.086909] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11950.174765] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [11955.021130] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [11955.176900] LustreError: 418275:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11955.197545] LustreError: 418275:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11959.272138] LustreError: 419295:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11959.285799] LustreError: 419295:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11963.775140] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11964.372629] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11969.342187] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [11977.995211] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11983.908919] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11991.015964] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11991.023821] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11991.047108] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11992.916541] Lustre: server umount lustre-OST0000 complete [11998.177493] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11998.195033] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [11998.203342] Lustre: Skipped 1 previous similar message [11998.208790] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11998.237127] Lustre: Skipped 1 previous similar message [12001.159405] Lustre: server umount lustre-MDT0000 complete [12003.301016] LustreError: 419302:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12003.316489] LustreError: 419302:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12005.417825] LustreError: 420043:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786755744 with bad export cookie 11881438024044539869 [12005.418819] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12005.432500] LustreError: 420043:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12015.344426] Lustre: server umount lustre-MDT0001 complete [12029.100460] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [12032.336282] Key type lgssc unregistered [12032.609515] LNet: 421834:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12032.625802] LNetError: 421834:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12032.661647] LNet: Removed LNI 192.168.206.151@tcp [12033.644815] Key type .llcrypt unregistered [12033.647646] Key type ._llcrypt unregistered [12060.228138] Key type ._llcrypt registered [12060.233967] Key type .llcrypt registered [12060.402687] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [12061.353976] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12061.367857] alg: No test for adler32 (adler32-zlib) [12062.519990] Lustre: Lustre: Build Version: 2.17.57_1_g59aa375 [12062.811786] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [12064.507045] Key type lgssc registered [12065.698516] Lustre: Echo OBD driver; http://www.lustre.org/ [12073.597918] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12073.623743] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12075.032575] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12080.167477] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [12085.224707] LustreError: 423253:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12085.242759] LustreError: 423253:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12088.272130] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12088.662800] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12092.830387] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [12099.913541] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12106.003391] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12112.222669] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12112.713111] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12118.769237] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [12119.921798] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [12127.136821] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12134.263872] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.51@tcp (stopping) [12134.880587] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12134.889768] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12134.901042] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12135.395532] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12135.412768] Lustre: Skipped 1 previous similar message [12137.773960] Lustre: server umount lustre-MDT0000 complete [12139.365919] LustreError: 424298:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12140.468942] LustreError: 423990:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.206.51@tcp arrived at 1786755879 with bad export cookie 6899994540609969142 [12140.482298] LustreError: 423990:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12140.514874] LustreError: 423254:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12140.537547] LustreError: 423254:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12142.317705] LustreError: 423235:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786755881 with bad export cookie 6899994540609968246 [12142.330262] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12144.617969] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12144.630183] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12144.651285] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12144.665202] Lustre: Skipped 3 previous similar messages [12145.632322] LustreError: 423253:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12145.651907] LustreError: 423253:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12147.683032] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12147.686866] Lustre: Skipped 1 previous similar message [12148.526405] Lustre: server umount lustre-OST0000 complete [12151.780629] LustreError: 424298:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12151.814676] LustreError: 424298:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [12155.324164] Lustre: server umount lustre-MDT0001 complete [12166.315937] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [12168.843985] Key type lgssc unregistered [12169.120611] LNet: 426941:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12169.125560] LNetError: 426941:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12169.139834] LNet: Removed LNI 192.168.206.151@tcp [12169.840330] Key type .llcrypt unregistered [12169.847168] Key type ._llcrypt unregistered [12194.575752] Key type ._llcrypt registered [12194.581635] Key type .llcrypt registered [12194.732551] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [12195.898790] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12196.017091] alg: No test for adler32 (adler32-zlib) [12197.063833] Lustre: Lustre: Build Version: 2.17.57_1_g59aa375 [12197.245583] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [12198.880301] Key type lgssc registered [12200.144955] Lustre: Echo OBD driver; http://www.lustre.org/ [12207.412053] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12207.454925] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12208.918890] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12213.519289] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [12219.369241] LustreError: 428359:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12219.377937] LustreError: 428359:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12221.320497] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12221.866204] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12227.128728] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [12235.467221] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12241.192795] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12247.729903] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12248.217587] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12254.081323] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [12254.387201] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [12261.095730] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12264.931795] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12264.938966] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12264.949461] Lustre: Skipped 1 previous similar message [12267.876189] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.51@tcp (stopping) [12267.889711] Lustre: Skipped 1 previous similar message [12270.051992] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12270.060107] Lustre: Skipped 1 previous similar message [12272.995940] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.51@tcp (stopping) [12278.118882] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.51@tcp (stopping) [12278.131073] Lustre: Skipped 2 previous similar messages [12279.263593] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12279.404328] Lustre: server umount lustre-MDT0001 complete [12280.295613] LustreError: 428355:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12285.409282] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12285.410663] LustreError: 429336:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12285.422833] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12285.454038] LustreError: 429336:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12290.542552] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12290.543599] LustreError: 428358:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12290.546526] Lustre: Skipped 1 previous similar message [12290.936856] Lustre: server umount lustre-OST0000 complete [12294.985954] Lustre: server umount lustre-MDT0000 complete [12308.512036] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [12311.659687] Key type lgssc unregistered [12311.924922] LNet: 432045:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12311.933326] LNetError: 432045:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12311.958738] LNet: Removed LNI 192.168.206.151@tcp [12312.685367] Key type .llcrypt unregistered [12312.688073] Key type ._llcrypt unregistered [12334.222632] Key type ._llcrypt registered [12334.226579] Key type .llcrypt registered [12334.328873] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_hostid [12349.383393] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [12350.429380] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12350.484364] alg: No test for adler32 (adler32-zlib) [12351.552336] Lustre: Lustre: Build Version: 2.17.57_1_g59aa375 [12351.851415] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [12353.487624] Key type lgssc registered [12354.793873] Lustre: Echo OBD driver; http://www.lustre.org/ [12393.780559] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [12413.265315] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12413.285428] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12414.553886] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12414.616846] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12414.736498] Lustre: lustre-MDT0000: new disk, initializing [12414.804462] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12414.832813] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12419.203949] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [12433.738609] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12433.977398] Lustre: lustre-OST0000: new disk, initializing [12433.983408] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12433.996712] Lustre: Skipped 1 previous similar message [12434.002962] Lustre: 437082:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12434.083214] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12440.015531] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [12440.149641] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [12440.162930] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [12440.197719] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [12448.279696] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12463.618263] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12463.728440] Lustre: 436064:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12463.779624] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12463.840477] Lustre: lustre-MDT0001: new disk, initializing [12463.952038] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12463.994925] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [12464.003019] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [12467.727452] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [12477.808879] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [12478.237530] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [12483.580764] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [12483.806587] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [12490.208366] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12490.219924] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12490.228766] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12490.238898] Lustre: Skipped 1 previous similar message [12490.254395] Lustre: Skipped 1 previous similar message [12491.156372] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.51@tcp (stopping) [12494.713578] Lustre: server umount lustre-MDT0001 complete [12500.449823] LustreError: 436072:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12500.470198] LustreError: 436072:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [12505.577969] LustreError: 436076:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12505.604093] LustreError: 436076:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12510.694163] LustreError: 436071:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12510.711121] LustreError: 436071:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12515.810338] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12515.813311] LustreError: 436076:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12515.824169] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12519.388899] Lustre: server umount lustre-OST0000 complete [12520.928175] LustreError: 436077:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12537.313095] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12537.625557] Lustre: server umount lustre-MDT0000 complete [12550.651291] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [12553.344709] Key type lgssc unregistered [12553.654150] LNet: 440837:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12553.666733] LNetError: 440837:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12553.679497] LNet: Removed LNI 192.168.206.151@tcp [12554.523258] Key type .llcrypt unregistered [12554.525750] Key type ._llcrypt unregistered [12589.657233] Key type ._llcrypt registered [12589.659745] Key type .llcrypt registered [12589.753633] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [12590.022527] Key type .llcrypt unregistered [12590.025968] Key type ._llcrypt unregistered [12605.525426] Key type ._llcrypt registered [12605.531423] Key type .llcrypt registered [12605.696215] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [12607.172537] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12607.248584] alg: No test for adler32 (adler32-zlib) [12608.269722] Lustre: Lustre: Build Version: 2.17.57_1_g59aa375 [12608.497229] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [12610.167338] Key type lgssc registered [12611.225970] Lustre: Echo OBD driver; http://www.lustre.org/ [12637.451570] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12637.504118] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12638.779056] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12638.816557] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12638.920288] Lustre: lustre-MDT0000: new disk, initializing [12639.037177] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12639.067313] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12644.245895] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [12654.641866] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12654.746369] Lustre: 444217:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12654.800973] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12654.804878] Lustre: Skipped 1 previous similar message [12654.965979] Lustre: lustre-MDT0001: new disk, initializing [12655.085666] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12655.145221] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12655.163386] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12659.916585] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [12669.418919] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12675.170502] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12681.648797] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12682.070310] Lustre: lustre-OST0000: new disk, initializing [12682.079547] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12682.096332] Lustre: 446182:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12682.230345] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12688.922182] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [12689.484078] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12689.510691] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12689.616190] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12699.259791] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12710.369994] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12710.378472] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12710.383612] Lustre: Skipped 1 previous similar message [12710.405409] Lustre: Skipped 2 previous similar messages [12713.250896] Lustre: server umount lustre-MDT0000 complete [12717.309484] LustreError: 446192:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786756456 with bad export cookie 1059478656155782668 [12717.311347] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12717.326941] LustreError: 446192:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12717.751930] Lustre: server umount lustre-MDT0001 complete [12731.067983] Lustre: server umount lustre-OST0000 complete [12740.683221] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 21:14:38 (1786756478) [12748.212110] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12748.639194] LustreError: 448210:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12748.658655] LustreError: 448210:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [12748.732365] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12753.172219] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [12753.891046] LustreError: 448211:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12757.986917] LustreError: 448210:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12761.231905] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12761.548751] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12765.673631] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [12772.951720] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12778.878855] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12798.514592] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12798.794608] Lustre: Found index 0 for lustre-OST0000, updating log [12798.986211] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12805.067903] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [12813.133630] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12819.508774] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [12829.668589] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12829.669875] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12829.670037] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12829.685461] Lustre: Skipped 2 previous similar messages [12830.754202] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12830.767702] Lustre: Skipped 2 previous similar messages [12835.184827] Lustre: server umount lustre-OST0000 complete [12851.212512] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12851.215291] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12851.226983] Lustre: Skipped 1 previous similar message [12851.239943] Lustre: Skipped 1 previous similar message [12856.292422] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12856.298328] Lustre: Skipped 1 previous similar message [12857.494238] Lustre: server umount lustre-MDT0000 complete [12858.349667] LustreError: 448207:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12858.362361] LustreError: 448207:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12858.399106] LNet: 1 local NIs in recovery (showing 1): 192.168.206.151@tcp [12861.114431] LustreError: 448191:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786756600 with bad export cookie 1059478656155784061 [12861.122345] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12861.128554] LustreError: 448191:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12861.528569] Lustre: server umount lustre-MDT0001 complete [12869.954072] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 21:16:47 (1786756607) [13490.029888] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13490.489504] LustreError: 454646:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13490.577124] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13495.636269] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [13495.786286] LustreError: 454647:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13499.872824] LustreError: 454646:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13505.216770] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13505.602377] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13510.741443] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [13519.143578] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13524.323036] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13544.680688] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13544.978350] Lustre: Found index 0 for lustre-OST0000, updating log [13545.342667] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13549.600174] LNet: 1 local NIs in recovery (showing 1): 192.168.206.151@tcp [13551.368622] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [13558.990461] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13561.042170] Lustre: server umount lustre-OST0000 complete [13565.602086] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13565.608336] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13565.613844] Lustre: Skipped 1 previous similar message [13565.629802] Lustre: Skipped 3 previous similar messages [13566.628757] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13566.633373] Lustre: Skipped 1 previous similar message [13568.672461] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13568.676510] Lustre: Skipped 3 previous similar messages [13570.885044] Lustre: server umount lustre-MDT0000 complete [13571.746720] LustreError: 454646:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13571.763830] LustreError: 454646:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13574.441726] LustreError: 454627:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786757313 with bad export cookie 1059478656155785146 [13574.442691] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13574.454233] LustreError: 454627:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13574.902444] Lustre: server umount lustre-MDT0001 complete [13603.934741] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_hostid [13608.991140] LNet: 1 local NIs in recovery (showing 1): 192.168.206.151@tcp [13613.564947] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [13647.045955] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 21:29:44 (1786757384) [13664.152331] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [13666.980200] Key type lgssc unregistered [13667.236808] LNet: 461261:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13667.245560] LNetError: 461261:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13797.288846] LNet: Removed LNI 192.168.206.151@tcp [13798.039299] Key type .llcrypt unregistered [13798.041809] Key type ._llcrypt unregistered [13811.823524] Key type ._llcrypt registered [13811.829108] Key type .llcrypt registered [13811.910971] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [13813.032430] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13813.060490] alg: No test for adler32 (adler32-zlib) [13814.042226] Lustre: Lustre: Build Version: 2.17.57_1_g59aa375 [13814.248680] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [13815.887130] Key type lgssc registered [13816.757916] Lustre: Echo OBD driver; http://www.lustre.org/ [13820.258259] LNet: Added LNI 192.168.206.151@tcp100 [8/256/0/180] [13821.081178] LNet: Added LNI 192.168.206.151@tcp101 [8/256/0/180] [13822.027311] LNet: Added LNI 192.168.206.151@tcp102 [8/256/0/180] [13822.893216] LNet: Added LNI 192.168.206.151@tcp103 [8/256/0/180] [13823.642473] LNet: Added LNI 192.168.206.151@tcp104 [8/256/0/180] [13824.445258] LNet: Added LNI 192.168.206.151@tcp105 [8/256/0/180] [13825.308180] LNet: Added LNI 192.168.206.151@tcp106 [8/256/0/180] [13826.134897] LNet: Added LNI 192.168.206.151@tcp107 [8/256/0/180] [13826.931927] LNet: Added LNI 192.168.206.151@tcp108 [8/256/0/180] [13827.705292] LNet: Added LNI 192.168.206.151@tcp109 [8/256/0/180] [13828.514484] LNet: Added LNI 192.168.206.151@tcp110 [8/256/0/180] [13829.220155] LNet: Added LNI 192.168.206.151@tcp111 [8/256/0/180] [13830.046838] LNet: Added LNI 192.168.206.151@tcp112 [8/256/0/180] [13830.791725] LNet: Added LNI 192.168.206.151@tcp113 [8/256/0/180] [13831.600704] LNet: Added LNI 192.168.206.151@tcp114 [8/256/0/180] [13832.376845] LNet: Added LNI 192.168.206.151@tcp115 [8/256/0/180] [13833.121442] LNet: Added LNI 192.168.206.151@tcp116 [8/256/0/180] [13833.867406] LNet: Added LNI 192.168.206.151@tcp117 [8/256/0/180] [13834.757037] LNet: Added LNI 192.168.206.151@tcp118 [8/256/0/180] [13835.559719] LNet: Added LNI 192.168.206.151@tcp119 [8/256/0/180] [13836.186812] LNet: Added LNI 192.168.206.151@tcp120 [8/256/0/180] [13837.082889] LNet: Added LNI 192.168.206.151@tcp121 [8/256/0/180] [13837.764430] LNet: Added LNI 192.168.206.151@tcp122 [8/256/0/180] [13838.492626] LNet: Added LNI 192.168.206.151@tcp123 [8/256/0/180] [13839.110053] LNet: Added LNI 192.168.206.151@tcp124 [8/256/0/180] [13839.910620] LNet: Added LNI 192.168.206.151@tcp125 [8/256/0/180] [13840.697374] LNet: Added LNI 192.168.206.151@tcp126 [8/256/0/180] [13841.492311] LNet: Added LNI 192.168.206.151@tcp127 [8/256/0/180] [13842.181528] LNet: Added LNI 192.168.206.151@tcp128 [8/256/0/180] [13842.986786] LNet: Added LNI 192.168.206.151@tcp129 [8/256/0/180] [13843.696635] LNet: Added LNI 192.168.206.151@tcp130 [8/256/0/180] [13844.384094] LNet: Added LNI 192.168.206.151@tcp131 [8/256/0/180] [13845.150762] LNet: Added LNI 192.168.206.151@tcp132 [8/256/0/180] [13845.840575] LNet: Added LNI 192.168.206.151@tcp133 [8/256/0/180] [13846.542823] LNet: Added LNI 192.168.206.151@tcp134 [8/256/0/180] [13847.185379] LNet: Added LNI 192.168.206.151@tcp135 [8/256/0/180] [13853.194523] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13853.209801] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13854.384215] Lustre: 464274:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.151@tcp: can't add NID 192.168.206.151@tcp130: rc = -75 [13854.483157] LNetError: 464296:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.151 [13854.510524] LNetError: Protocol error connecting to 192.168.206.151@tcp112 at host 192.168.206.151:988: is it running a compatible version of Lustre? [13854.533753] LNetError: Skipped 5 previous similar messages [13854.664075] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13854.706880] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13854.810780] Lustre: lustre-MDT0000: new disk, initializing [13854.889919] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13854.910837] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13855.199393] LNet: 5 local NIs in recovery (showing 5): 192.168.206.151@tcp126, 192.168.206.151@tcp125, 192.168.206.151@tcp127, 192.168.206.151@tcp128, 192.168.206.151@tcp129 [13855.203064] LNetError: 461733:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.151 [13855.225769] LNetError: 461733:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 7 previous similar messages [13855.235211] LNetError: Protocol error connecting to 192.168.206.151@tcp111 at host 192.168.206.151:988: is it running a compatible version of Lustre? [13855.240590] LNetError: Skipped 2 previous similar messages [13859.347225] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [13870.916755] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13871.025864] Lustre: 464290:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13871.079751] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13871.088026] Lustre: Skipped 1 previous similar message [13871.186517] Lustre: lustre-MDT0001: new disk, initializing [13871.228192] Lustre: 465093:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.151@tcp: can't add NID 192.168.206.151@tcp130: rc = -75 [13871.235255] Lustre: 465093:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [13871.362685] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13871.399659] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13871.414462] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13874.325897] Lustre: 465257:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.151@tcp: can't add NID 192.168.206.151@tcp130: rc = -75 [13874.342425] Lustre: 465257:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [13876.384624] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [13885.449599] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13885.920401] LNetError: 464296:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.151 [13885.930366] LNetError: 464296:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 2 previous similar messages [13885.940230] LNetError: Protocol error connecting to 192.168.206.151@tcp123 at host 192.168.206.151:988: is it running a compatible version of Lustre? [13885.953831] LNetError: Skipped 1 previous similar message [13890.577737] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13891.040930] LNetError: 464295:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.151 [13891.050097] LNetError: 464295:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 1 previous similar message [13891.070641] LNetError: Protocol error connecting to 192.168.206.151@tcp111 at host 192.168.206.151:988: is it running a compatible version of Lustre? [13891.089644] LNetError: Skipped 1 previous similar message [13896.165983] LNetError: 461731:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.151 [13896.175717] LNetError: 461731:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 2 previous similar messages [13896.181842] LNetError: Protocol error connecting to 192.168.206.151@tcp112 at host 192.168.206.151:988: is it running a compatible version of Lustre? [13896.190439] LNetError: Skipped 2 previous similar messages [13897.674049] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13898.072672] Lustre: lustre-OST0000: new disk, initializing [13898.076706] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13898.084697] Lustre: 466265:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13898.181995] Lustre: 466270:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.151@tcp: can't add NID 192.168.206.151@tcp130: rc = -75 [13898.193894] Lustre: 466270:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [13898.265676] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13905.263969] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [13905.378204] LNetError: 464294:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.151 [13905.393247] LNetError: 464294:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 3 previous similar messages [13905.405578] LNetError: Protocol error connecting to 192.168.206.151@tcp127 at host 192.168.206.151:988: is it running a compatible version of Lustre? [13905.417908] LNetError: Skipped 3 previous similar messages [13905.939608] Lustre: 466645:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.151@tcp: can't add NID 192.168.206.151@tcp130: rc = -75 [13905.952387] Lustre: 466645:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [13905.992473] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13906.001160] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13906.086829] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13916.216095] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13921.763320] LNetError: 464300:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.151 [13921.779919] LNetError: 464300:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 7 previous similar messages [13921.792177] LNetError: Protocol error connecting to 192.168.206.151@tcp112 at host 192.168.206.151:988: is it running a compatible version of Lustre? [13921.805638] LNetError: Skipped 7 previous similar messages [13926.884644] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13926.894303] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13926.896193] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13926.908552] Lustre: Skipped 1 previous similar message [13926.922254] Lustre: Skipped 1 previous similar message [13932.001657] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13932.013804] Lustre: Skipped 1 previous similar message [13937.126888] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13937.138164] Lustre: Skipped 1 previous similar message [13938.143138] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13938.427266] Lustre: server umount lustre-OST0000 complete [13942.257371] LustreError: 461742:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000a3c96ca2 x1873551087936128/t0(0) o8->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13942.266794] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13942.292878] Lustre: Skipped 1 previous similar message [13942.295865] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13942.300571] Lustre: Skipped 1 previous similar message [13947.361679] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13947.369329] Lustre: Skipped 1 previous similar message [13948.618853] Lustre: server umount lustre-MDT0000 complete [13952.113668] LustreError: 464284:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786757691 with bad export cookie 17861599523493526361 [13952.117418] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13952.129712] LustreError: 464284:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13952.527133] Lustre: server umount lustre-MDT0001 complete [13956.579585] LNetError: 461733:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.151 [13956.590691] LNetError: 461733:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 6 previous similar messages [13956.601778] LNetError: Protocol error connecting to 192.168.206.151@tcp127 at host 192.168.206.151:988: is it running a compatible version of Lustre? [13956.617560] LNetError: Skipped 6 previous similar messages [13966.810472] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [13971.284645] Key type lgssc unregistered [13971.808676] LNet: 468186:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13971.825580] LNetError: 468186:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13971.834062] LNet: Removed LNI 192.168.206.151@tcp [13972.895158] LNet: Removed LNI 192.168.206.151@tcp100 [13972.900587] LNetError: 468186:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13972.909232] LNetError: 468186:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [13972.916295] LNet: Removed LNI 192.168.206.151@tcp101 [13973.983629] LNet: Removed LNI 192.168.206.151@tcp102 [13973.985933] LNetError: 468186:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13973.994728] LNetError: 468186:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [13974.001207] LNet: Removed LNI 192.168.206.151@tcp103 [13975.007206] LNet: Removed LNI 192.168.206.151@tcp104 [13976.032670] LNet: Removed LNI 192.168.206.151@tcp105 [13976.047453] LNetError: 468186:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13976.057391] LNetError: 468186:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [13976.067222] LNet: Removed LNI 192.168.206.151@tcp106 [13977.119212] LNet: Removed LNI 192.168.206.151@tcp107 [13978.143166] LNet: Removed LNI 192.168.206.151@tcp108 [13979.168497] LNet: Removed LNI 192.168.206.151@tcp109 [13980.191169] LNet: Removed LNI 192.168.206.151@tcp110 [13980.193085] LNetError: 468186:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13980.205132] LNetError: 468186:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [13980.212291] LNet: Removed LNI 192.168.206.151@tcp111 [13981.279137] LNet: Removed LNI 192.168.206.151@tcp112 [13982.303156] LNet: Removed LNI 192.168.206.151@tcp113 [13983.327148] LNet: Removed LNI 192.168.206.151@tcp114 [13984.351152] LNet: Removed LNI 192.168.206.151@tcp115 [13985.380977] LNet: Removed LNI 192.168.206.151@tcp116 [13986.399856] LNet: Removed LNI 192.168.206.151@tcp117 [13987.423669] LNet: Removed LNI 192.168.206.151@tcp118 [13988.447207] LNet: Removed LNI 192.168.206.151@tcp119 [13988.460072] LNetError: 468186:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13988.466423] LNetError: 468186:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [13988.490554] LNet: Removed LNI 192.168.206.151@tcp120 [13989.535159] LNet: Removed LNI 192.168.206.151@tcp121 [13990.559159] LNet: Removed LNI 192.168.206.151@tcp122 [13991.584187] LNet: Removed LNI 192.168.206.151@tcp123 [13992.607164] LNet: Removed LNI 192.168.206.151@tcp124 [13993.631314] LNet: Removed LNI 192.168.206.151@tcp125 [13994.657413] LNet: Removed LNI 192.168.206.151@tcp126 [13995.679217] LNet: Removed LNI 192.168.206.151@tcp127 [13996.703194] LNet: Removed LNI 192.168.206.151@tcp128 [13997.727315] LNet: Removed LNI 192.168.206.151@tcp129 [13997.732534] LNet: Removed LNI 192.168.206.151@tcp130 [13997.736535] LNet: Removed LNI 192.168.206.151@tcp131 [13997.740192] LNet: Removed LNI 192.168.206.151@tcp132 [13997.743265] LNet: Removed LNI 192.168.206.151@tcp133 [13997.749028] LNet: Removed LNI 192.168.206.151@tcp134 [13997.765123] LNet: Removed LNI 192.168.206.151@tcp135 [13998.534919] Key type .llcrypt unregistered [13998.542576] Key type ._llcrypt unregistered [14018.718289] Key type ._llcrypt registered [14018.721389] Key type .llcrypt registered [14018.822341] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_hostid [14033.298106] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [14034.041679] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14034.072720] alg: No test for adler32 (adler32-zlib) [14035.077448] Lustre: Lustre: Build Version: 2.17.57_1_g59aa375 [14035.231858] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [14036.871154] Key type lgssc registered [14037.786920] Lustre: Echo OBD driver; http://www.lustre.org/ [14071.022703] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 21:36:48 (1786757808) [14088.871338] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [14091.209744] Key type lgssc unregistered [14091.470188] LNet: 472274:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14091.484296] LNetError: 472274:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14091.508679] LNet: Removed LNI 192.168.206.151@tcp [14092.211488] Key type .llcrypt unregistered [14092.216758] Key type ._llcrypt unregistered [14107.379945] Key type ._llcrypt registered [14107.384839] Key type .llcrypt registered [14107.516472] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [14108.464970] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14108.493937] alg: No test for adler32 (adler32-zlib) [14109.614247] Lustre: Lustre: Build Version: 2.17.57_1_g59aa375 [14109.857966] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [14111.567279] Key type lgssc registered [14112.985749] Lustre: Echo OBD driver; http://www.lustre.org/ [14123.106153] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14123.129563] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14124.493476] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14124.525556] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14124.641400] Lustre: lustre-MDT0000: new disk, initializing [14124.731722] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14124.747653] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14129.275073] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [14140.491480] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14140.617153] Lustre: 473482:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14140.709150] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14140.716983] Lustre: Skipped 1 previous similar message [14140.834916] Lustre: lustre-MDT0001: new disk, initializing [14140.961166] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14141.030469] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14141.045688] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14145.787896] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [14154.726646] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14159.996417] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14166.825489] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14167.170161] Lustre: lustre-OST0000: new disk, initializing [14167.175247] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14167.187264] Lustre: 475453:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14167.278899] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14173.365891] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [14176.343728] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14176.349726] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14176.525731] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14182.546991] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14186.291307] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing lnet_if_list [14191.447648] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [14191.473924] LNet: Added LNI 192.168.206.151@tcp10 [8/256/0/180] [14196.614847] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [14196.674780] LNet: Added LNI 192.168.206.151@tcp11 [8/256/0/180] [14200.693884] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [14200.756286] LNet: Added LNI 192.168.206.151@tcp12 [8/256/0/180] [14205.039579] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [14205.071758] LNet: Added LNI 192.168.206.151@tcp13 [8/256/0/180] [14209.632253] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [14209.653896] LNet: Added LNI 192.168.206.151@tcp14 [8/256/0/180] [14213.875782] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [14213.897408] LNet: Added LNI 192.168.206.151@tcp15 [8/256/0/180] [14218.499867] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [14218.541680] LNet: Added LNI 192.168.206.151@tcp16 [8/256/0/180] [14223.972977] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [14223.999211] LNet: Added LNI 192.168.206.151@tcp17 [8/256/0/180] [14229.192847] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [14229.218403] LNet: Added LNI 192.168.206.151@tcp18 [8/256/0/180] [14235.085683] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [14235.154044] LNet: Added LNI 192.168.206.151@tcp19 [8/256/0/180] [14240.624919] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [14240.688333] LNet: Added LNI 192.168.206.151@tcp20 [8/256/0/180] [14246.119352] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [14246.148188] LNet: Added LNI 192.168.206.151@tcp21 [8/256/0/180] [14251.953762] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [14251.982725] LNet: Added LNI 192.168.206.151@tcp22 [8/256/0/180] [14256.836085] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [14256.865394] LNet: Added LNI 192.168.206.151@tcp23 [8/256/0/180] [14262.610351] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [14262.643183] LNet: Added LNI 192.168.206.151@tcp24 [8/256/0/180] [14268.269886] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [14268.318693] LNet: Added LNI 192.168.206.151@tcp25 [8/256/0/180] [14273.512267] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [14273.533765] LNet: Added LNI 192.168.206.151@tcp26 [8/256/0/180] [14278.460858] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [14278.480975] LNet: Added LNI 192.168.206.151@tcp27 [8/256/0/180] [14284.317962] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [14284.360228] LNet: Added LNI 192.168.206.151@tcp28 [8/256/0/180] [14289.723650] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [14289.763427] LNet: Added LNI 192.168.206.151@tcp29 [8/256/0/180] [14295.586621] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [14295.613892] LNet: Added LNI 192.168.206.151@tcp30 [8/256/0/180] [14301.472115] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [14301.493259] LNet: Added LNI 192.168.206.151@tcp31 [8/256/0/180] [14307.657254] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [14307.682038] LNet: Added LNI 192.168.206.151@tcp32 [8/256/0/180] [14314.212737] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [14314.247879] LNet: Added LNI 192.168.206.151@tcp33 [8/256/0/180] [14319.895983] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [14319.936815] LNet: Added LNI 192.168.206.151@tcp34 [8/256/0/180] [14325.633320] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [14325.654225] LNet: Added LNI 192.168.206.151@tcp35 [8/256/0/180] [14332.893232] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [14332.921736] LNet: Added LNI 192.168.206.151@tcp36 [8/256/0/180] [14339.908854] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [14339.929596] LNet: Added LNI 192.168.206.151@tcp37 [8/256/0/180] [14347.035736] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [14347.069928] LNet: Added LNI 192.168.206.151@tcp38 [8/256/0/180] [14354.726158] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [14354.757494] LNet: Added LNI 192.168.206.151@tcp39 [8/256/0/180] [14362.122629] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [14362.150552] LNet: Added LNI 192.168.206.151@tcp40 [8/256/0/180] [14369.114255] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [14369.140744] LNet: Added LNI 192.168.206.151@tcp41 [8/256/0/180] [14376.845085] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [14376.877186] LNet: Added LNI 192.168.206.151@tcp42 [8/256/0/180] [14385.355487] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [14385.383887] LNet: Added LNI 192.168.206.151@tcp43 [8/256/0/180] [14393.330819] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [14393.388060] LNet: Added LNI 192.168.206.151@tcp44 [8/256/0/180] [14401.573951] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [14401.613938] LNet: Added LNI 192.168.206.151@tcp45 [8/256/0/180] [14412.021305] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [14412.079102] LNet: Added LNI 192.168.206.151@tcp46 [8/256/0/180] [14419.893810] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [14419.942265] LNet: Added LNI 192.168.206.151@tcp47 [8/256/0/180] [14427.834401] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [14427.891751] LNet: Added LNI 192.168.206.151@tcp48 [8/256/0/180] [14435.704884] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [14435.734161] LNet: Added LNI 192.168.206.151@tcp49 [8/256/0/180] [14444.657530] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [14444.687825] LNet: Added LNI 192.168.206.151@tcp50 [8/256/0/180] [14463.466662] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14463.473982] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14463.491273] Lustre: Skipped 2 previous similar messages [14463.492318] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14463.519153] Lustre: Skipped 2 previous similar messages [14468.576250] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14468.580606] Lustre: Skipped 2 previous similar messages [14473.698534] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14473.712641] Lustre: Skipped 2 previous similar messages [14476.255500] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14476.438446] Lustre: server umount lustre-MDT0000 complete [14479.832704] LustreError: 473474:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786758219 with bad export cookie 927727047419286013 [14479.835835] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14479.852604] LustreError: 473474:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14480.185626] Lustre: server umount lustre-MDT0001 complete [14499.554907] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [14500.321488] Lustre: 472758:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786758223/real 1786758223] req@000000006941cb36 x1873551398437888/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786758239 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14500.340689] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14501.345179] Lustre: 472757:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786758224/real 1786758224] req@0000000028156817 x1873551398438016/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786758240 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14505.443914] Lustre: 472755:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786758228/real 1786758228] req@00000000d3028534 x1873551398438144/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786758244 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14510.063733] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14511.583963] Lustre: 472758:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786758234/real 1786758234] req@0000000017544fde x1873551398438528/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786758250 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14511.638571] Lustre: 472758:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [14516.703177] Lustre: 472757:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786758239/real 1786758239] req@00000000e68c7dda x1873551398438656/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786758255 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14524.985060] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xcdff2d595f2dc29 [14524.998755] Lustre: MGC192.168.206.151@tcp: Connection restored to 0@lo (at 0@lo) [14525.415260] LustreError: 475445:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14525.441645] LustreError: 475445:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [14525.501415] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14529.504672] LustreError: 475446:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14530.335162] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [14534.628937] LustreError: 475446:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14534.630156] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14539.361287] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14539.618071] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14543.995516] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [14545.895657] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14551.573446] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14558.202634] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14585.826885] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14585.835524] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14585.860958] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14586.853162] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14589.422847] Lustre: server umount lustre-OST0000 complete [14596.070229] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14596.079673] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14596.093495] Lustre: Skipped 1 previous similar message [14596.101359] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14596.106463] Lustre: Skipped 2 previous similar messages [14600.924630] Lustre: server umount lustre-MDT0000 complete [14602.212321] LustreError: 499695:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14602.234429] LustreError: 499695:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [14605.052136] LustreError: 475452:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786758344 with bad export cookie 927727047419288617 [14605.058811] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14605.084950] LustreError: 475452:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14605.595700] Lustre: server umount lustre-MDT0001 complete [14618.668128] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [14621.939445] Key type lgssc unregistered [14622.561302] LNet: 502499:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14622.587563] LNetError: 502499:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14622.596843] LNet: Removed LNI 192.168.206.151@tcp [14622.604577] LNet: Removed LNI 192.168.206.151@tcp10 [14622.611553] LNet: Removed LNI 192.168.206.151@tcp11 [14622.617569] LNet: Removed LNI 192.168.206.151@tcp12 [14622.623986] LNet: Removed LNI 192.168.206.151@tcp13 [14622.635472] LNet: Removed LNI 192.168.206.151@tcp14 [14622.647560] LNet: Removed LNI 192.168.206.151@tcp15 [14622.653952] LNet: Removed LNI 192.168.206.151@tcp16 [14622.661830] LNet: Removed LNI 192.168.206.151@tcp17 [14622.673367] LNet: Removed LNI 192.168.206.151@tcp18 [14622.675322] LNet: Removed LNI 192.168.206.151@tcp19 [14622.692250] LNet: Removed LNI 192.168.206.151@tcp20 [14622.697899] LNet: Removed LNI 192.168.206.151@tcp21 [14622.714108] LNet: Removed LNI 192.168.206.151@tcp22 [14622.726735] LNet: Removed LNI 192.168.206.151@tcp23 [14622.728779] LNet: Removed LNI 192.168.206.151@tcp24 [14622.740559] LNet: Removed LNI 192.168.206.151@tcp25 [14622.748469] LNet: Removed LNI 192.168.206.151@tcp26 [14622.759542] LNet: Removed LNI 192.168.206.151@tcp27 [14622.761606] LNet: Removed LNI 192.168.206.151@tcp28 [14622.764943] LNet: Removed LNI 192.168.206.151@tcp29 [14622.772545] LNet: Removed LNI 192.168.206.151@tcp30 [14622.779645] LNet: Removed LNI 192.168.206.151@tcp31 [14622.788880] LNet: Removed LNI 192.168.206.151@tcp32 [14622.798730] LNet: Removed LNI 192.168.206.151@tcp33 [14622.800805] LNet: Removed LNI 192.168.206.151@tcp34 [14622.813750] LNet: Removed LNI 192.168.206.151@tcp35 [14622.837994] LNet: Removed LNI 192.168.206.151@tcp36 [14622.844472] LNet: Removed LNI 192.168.206.151@tcp37 [14622.852536] LNet: Removed LNI 192.168.206.151@tcp38 [14622.858962] LNet: Removed LNI 192.168.206.151@tcp39 [14622.877095] LNet: Removed LNI 192.168.206.151@tcp40 [14622.892090] LNet: Removed LNI 192.168.206.151@tcp41 [14622.901575] LNet: Removed LNI 192.168.206.151@tcp42 [14622.906229] LNet: Removed LNI 192.168.206.151@tcp43 [14622.909983] LNet: Removed LNI 192.168.206.151@tcp44 [14622.913240] LNet: Removed LNI 192.168.206.151@tcp45 [14622.922620] LNet: Removed LNI 192.168.206.151@tcp46 [14622.930633] LNet: Removed LNI 192.168.206.151@tcp47 [14622.937224] LNet: Removed LNI 192.168.206.151@tcp48 [14622.949389] LNet: Removed LNI 192.168.206.151@tcp49 [14622.967954] LNet: Removed LNI 192.168.206.151@tcp50 [14624.132783] Key type .llcrypt unregistered [14624.134525] Key type ._llcrypt unregistered [14651.784375] Key type ._llcrypt registered [14651.786157] Key type .llcrypt registered [14651.912664] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_hostid [14668.700363] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [14669.893165] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14670.117956] alg: No test for adler32 (adler32-zlib) [14671.282268] Lustre: Lustre: Build Version: 2.17.57_1_g59aa375 [14671.569283] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [14673.232115] Key type lgssc registered [14674.734694] Lustre: Echo OBD driver; http://www.lustre.org/ [14717.242417] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14717.286150] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14718.538079] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14718.560807] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14718.638156] Lustre: lustre-MDT0000: new disk, initializing [14718.703058] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14718.734193] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14723.405295] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [14732.906945] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14733.050420] Lustre: 505714:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14733.101668] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14733.105263] Lustre: Skipped 1 previous similar message [14733.171256] Lustre: lustre-MDT0001: new disk, initializing [14733.266798] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14733.305000] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14733.314183] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14737.633081] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [14747.608567] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14754.170599] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14762.060139] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14762.293525] Lustre: *** cfs_fail_loc=90f, val=0*** [14762.401806] Lustre: lustre-OST0000: new disk, initializing [14762.408121] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14762.413280] Lustre: 507728:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14762.527523] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14768.474618] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [14770.720420] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14770.732120] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14770.808940] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14777.858239] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14786.530707] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14786.540792] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14786.552312] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14791.137045] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14791.155888] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14791.178619] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14796.780162] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14796.788077] Lustre: Skipped 1 previous similar message [14800.352316] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14800.540443] Lustre: server umount lustre-OST0000 complete [14804.961176] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14804.971818] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14804.983169] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14804.987924] Lustre: Skipped 1 previous similar message [14807.009937] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14810.386510] Lustre: server umount lustre-MDT0000 complete [14813.857092] LustreError: 505704:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786758553 with bad export cookie 17083362184345342693 [14813.858791] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14813.877746] LustreError: 505704:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14814.164487] Lustre: server umount lustre-MDT0001 complete [14823.803500] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [14826.629860] Key type lgssc unregistered [14826.871853] LNet: 509637:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14826.875400] LNetError: 509637:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14827.952200] LNet: Removed LNI 192.168.206.151@tcp [14828.684543] Key type .llcrypt unregistered [14828.686648] Key type ._llcrypt unregistered [14846.924872] Key type ._llcrypt registered [14846.929767] Key type .llcrypt registered [14847.035362] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [14847.997821] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14848.094918] alg: No test for adler32 (adler32-zlib) [14849.104340] Lustre: Lustre: Build Version: 2.17.57_1_g59aa375 [14849.360085] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [14851.015915] Key type lgssc registered [14852.265521] Lustre: Echo OBD driver; http://www.lustre.org/ [14860.229720] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14860.245924] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14861.806067] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14866.095112] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [14872.054362] LustreError: 510908:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14872.081209] LustreError: 510908:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [14874.515820] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14875.001440] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14879.675224] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [14887.506096] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14893.323340] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14900.982886] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14901.498722] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14908.428335] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [14915.720297] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14929.357734] loop0: detected capacity change from 0 to 204800000 [14929.554586] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14929.575568] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14929.587890] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14934.956734] loop0: detected capacity change from 0 to 204800000 [14935.107040] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14935.116278] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14935.130244] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14939.734276] loop0: detected capacity change from 0 to 204800000 [14944.854047] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14945.033792] Lustre: fs15246-OST0000: new disk, initializing [14945.043792] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [14945.050588] Lustre: 514499:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14945.145316] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14951.358755] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [14960.947500] loop1: detected capacity change from 0 to 204800000 [14966.300783] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14966.383321] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [14966.475643] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [14966.577696] Lustre: fs15246-MDT0000: new disk, initializing [14966.737590] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14966.760357] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14966.856605] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [14967.905920] Lustre: 515622:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@000000001b579b8a x1873552173218560/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1786758723 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [14971.314868] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [14972.976705] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [14972.994012] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [14973.037704] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [14992.895368] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14992.916342] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [14997.989533] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [14998.227719] Lustre: server umount fs15246-MDT0000 complete [15003.107039] LustreError: 510902:0:(ldlm_lib.c:1192:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15007.709816] Lustre: server umount fs15246-OST0000 complete [15020.514909] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15020.519333] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15020.534318] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15022.560879] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15022.568568] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15022.584669] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15025.749871] Lustre: server umount lustre-OST0000 complete [15033.826666] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15033.827197] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15033.846262] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15033.865356] Lustre: Skipped 3 previous similar messages [15035.731474] Lustre: server umount lustre-MDT0000 complete [15038.951890] LustreError: 514527:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15038.976756] LustreError: 514527:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [15039.417919] LustreError: 510889:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786758778 with bad export cookie 11438947291186907792 [15039.424563] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15039.433284] LustreError: 510889:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15039.711283] Lustre: server umount lustre-MDT0001 complete [15050.743961] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [15053.888781] Key type lgssc unregistered [15054.207882] LNet: 518722:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15054.216792] LNetError: 518722:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15054.232863] LNet: Removed LNI 192.168.206.151@tcp [15055.080782] Key type .llcrypt unregistered [15055.082933] Key type ._llcrypt unregistered [15074.083828] Key type ._llcrypt registered [15074.086808] Key type .llcrypt registered [15074.192886] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [15075.087929] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15075.159659] alg: No test for adler32 (adler32-zlib) [15076.363660] Lustre: Lustre: Build Version: 2.17.57_1_g59aa375 [15076.602994] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [15078.256244] Key type lgssc registered [15079.298847] Lustre: Echo OBD driver; http://www.lustre.org/ [15115.921254] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_hostid [15125.449353] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [15156.933639] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15156.982092] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15158.248450] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15158.284787] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15158.373879] Lustre: lustre-MDT0000: new disk, initializing [15158.444497] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15158.459365] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15162.773431] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [15173.189708] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15173.365916] Lustre: 523190:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15173.476162] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15173.483741] Lustre: Skipped 1 previous similar message [15173.595290] Lustre: lustre-MDT0001: new disk, initializing [15173.705680] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15173.744079] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15173.759381] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15178.171308] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [15187.415469] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15193.184324] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15200.751990] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15201.102705] Lustre: lustre-OST0000: new disk, initializing [15201.107458] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15201.117131] Lustre: 525157:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15201.221159] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15206.833438] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [15209.596815] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15209.610069] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15209.772624] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15216.212797] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15220.787223] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [15220.987526] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [15225.085063] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [15225.279385] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [15230.434260] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15230.437269] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15230.456095] LustreError: Skipped 1 previous similar message [15230.457136] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15230.499094] Lustre: Skipped 1 previous similar message [15233.394624] Lustre: server umount lustre-OST0000 complete [15240.167229] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15240.175182] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15240.185814] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15240.188904] Lustre: Skipped 1 previous similar message [15243.268163] Lustre: server umount lustre-MDT0000 complete [15247.121601] LustreError: 523181:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786758986 with bad export cookie 656041293464871142 [15247.122413] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15247.141139] LustreError: 523181:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [15247.660551] Lustre: server umount lustre-MDT0001 complete [15276.714451] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_hostid [15287.546257] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [15325.269739] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 15047 sec ========= 21:57:42 (1786759062) [15327.106670] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 21:57:44 (1786759064) === [15330.139542] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 21:57:47 (1786759067) === [15371.051232] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [15373.985381] Key type lgssc unregistered [15374.340895] LNet: 531806:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15374.354812] LNetError: 531806:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15374.364695] LNet: Removed LNI 192.168.206.151@tcp [15375.161049] Key type .llcrypt unregistered [15375.164564] Key type ._llcrypt unregistered