[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 708291712 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.002379] x2apic enabled [ 0.004004] Switched APIC routing to physical x2apic. [ 0.005015] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008020] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009011] pid_max: default: 32768 minimum: 301 [ 0.010139] LSM: Security Framework initializing [ 0.011062] Yama: becoming mindful. [ 0.012063] SELinux: Initializing. [ 0.013081] *** VALIDATE selinux *** [ 0.022410] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027814] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028153] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029109] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030105] *** VALIDATE tmpfs *** [ 0.031444] *** VALIDATE proc *** [ 0.032000] *** VALIDATE cgroup *** [ 0.032013] *** VALIDATE cgroup2 *** [ 0.033267] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.034160] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.035008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.036023] Spectre V2 : User space: Vulnerable [ 0.037008] Speculative Store Bypass: Vulnerable [ 0.040055] debug: unmapping init [mem 0xffffffffb9259000-0xffffffffb9260fff] [ 0.042154] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.043693] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.044021] ... version: 2 [ 0.045012] ... bit width: 48 [ 0.046011] ... generic registers: 4 [ 0.047010] ... value mask: 0000ffffffffffff [ 0.048014] ... max period: 00007fffffffffff [ 0.049011] ... fixed-purpose events: 3 [ 0.050008] ... event mask: 000000070000000f [ 0.051283] rcu: Hierarchical SRCU implementation. [ 0.053463] smp: Bringing up secondary CPUs ... [ 0.055449] x86: Booting SMP configuration: [ 0.056026] .... node #0, CPUs: #1 #2 #3 [ 0.066466] smp: Brought up 1 node, 4 CPUs [ 0.068019] smpboot: Max logical packages: 1 [ 0.069098] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.106844] node 0 deferred pages initialised in 36ms [ 0.111216] devtmpfs: initialized [ 0.112296] x86/mm: Memory block size: 128MB [ 0.115047] gcov: version magic: 0x41383552 [ 0.117225] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.118066] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.119254] pinctrl core: initialized pinctrl subsystem [ 0.120353] [ 0.120968] ************************************************************* [ 0.121011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.122011] ** ** [ 0.123009] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.124013] ** ** [ 0.125010] ** This means that this kernel is built to expose internal ** [ 0.126013] ** IOMMU data structures, which may compromise security on ** [ 0.127013] ** your system. ** [ 0.128014] ** ** [ 0.129010] ** If you see this message and you are not debugging the ** [ 0.130014] ** kernel, report this immediately to your vendor! ** [ 0.131012] ** ** [ 0.132011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.133013] ************************************************************* [ 0.134933] NET: Registered protocol family 16 [ 0.135420] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.136051] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.137054] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.138639] cpuidle: using governor menu [ 0.139727] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.142690] PCI: Using configuration type 1 for base access [ 0.145121] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.155404] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.157018] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.161094] cryptd: max_cpu_qlen set to 1000 [ 0.163226] ACPI: Added _OSI(Module Device) [ 0.164011] ACPI: Added _OSI(Processor Device) [ 0.165009] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.166009] ACPI: Added _OSI(Processor Aggregator Device) [ 0.170511] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.174394] ACPI: Interpreter enabled [ 0.175058] ACPI: PM: (supports S0 S3 S4 S5) [ 0.176011] ACPI: Using IOAPIC for interrupt routing [ 0.177088] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.178420] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.189222] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.190049] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.191019] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.192076] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.194433] acpiphp: Slot [2] registered [ 0.195095] acpiphp: Slot [5] registered [ 0.196096] acpiphp: Slot [6] registered [ 0.197096] acpiphp: Slot [7] registered [ 0.198140] acpiphp: Slot [8] registered [ 0.199176] acpiphp: Slot [9] registered [ 0.200190] acpiphp: Slot [10] registered [ 0.201111] acpiphp: Slot [3] registered [ 0.202068] acpiphp: Slot [4] registered [ 0.203084] acpiphp: Slot [11] registered [ 0.204082] acpiphp: Slot [12] registered [ 0.205079] acpiphp: Slot [13] registered [ 0.206104] acpiphp: Slot [14] registered [ 0.207081] acpiphp: Slot [15] registered [ 0.208086] acpiphp: Slot [16] registered [ 0.209164] acpiphp: Slot [17] registered [ 0.210094] acpiphp: Slot [18] registered [ 0.211100] acpiphp: Slot [19] registered [ 0.212000] acpiphp: Slot [20] registered [ 0.213090] acpiphp: Slot [21] registered [ 0.215142] acpiphp: Slot [22] registered [ 0.216109] acpiphp: Slot [23] registered [ 0.218091] acpiphp: Slot [24] registered [ 0.220079] acpiphp: Slot [25] registered [ 0.221077] acpiphp: Slot [26] registered [ 0.222070] acpiphp: Slot [27] registered [ 0.223096] acpiphp: Slot [28] registered [ 0.225105] acpiphp: Slot [29] registered [ 0.227089] acpiphp: Slot [30] registered [ 0.228091] acpiphp: Slot [31] registered [ 0.230134] PCI host bridge to bus 0000:00 [ 0.232015] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.235020] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.237017] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.239018] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.242025] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.246022] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.248211] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.251047] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.254440] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.266013] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.271000] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.272019] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.274016] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.277015] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.279662] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.282896] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.286036] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.289947] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.295014] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.309014] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.316014] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.320939] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.334016] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.348000] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.382000] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.393000] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.407014] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.419014] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.459016] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.472697] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.486013] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.496012] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.526014] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.537207] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.545015] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.552014] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.568015] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.581850] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.590013] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.600022] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.628015] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.642827] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.656012] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.663015] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.682015] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.691043] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.693333] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.697358] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.701444] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.704228] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.709207] iommu: Default domain type: Passthrough [ 0.713379] SCSI subsystem initialized [ 0.716169] ACPI: bus type USB registered [ 0.719426] usbcore: registered new interface driver usbfs [ 0.724463] usbcore: registered new interface driver hub [ 0.728084] usbcore: registered new device driver usb [ 0.733184] pps_core: LinuxPPS API ver. 1 registered [ 0.736010] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.742057] PTP clock support registered [ 0.748188] EDAC MC: Ver: 3.0.0 [ 0.751008] PCI: Using ACPI for IRQ routing [ 0.752253] NetLabel: Initializing [ 0.755012] NetLabel: domain hash size = 128 [ 0.758010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.761078] NetLabel: unlabeled traffic allowed by default [ 0.765522] vgaarb: loaded [ 0.767834] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.771009] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.777678] clocksource: Switched to clocksource kvm-clock [ 0.932641] VFS: Disk quotas dquot_6.6.0 [ 0.933948] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.938109] *** VALIDATE ramfs *** [ 0.939592] *** VALIDATE hugetlbfs *** [ 0.942649] pnp: PnP ACPI init [ 0.944927] pnp: PnP ACPI: found 6 devices [ 0.964661] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.967902] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.970386] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.972409] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.974580] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.976731] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.979579] NET: Registered protocol family 2 [ 0.981866] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.986863] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.990022] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.994751] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.997873] TCP: Hash tables configured (established 65536 bind 65536) [ 1.000877] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.003534] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.005833] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.008721] NET: Registered protocol family 1 [ 1.011958] RPC: Registered named UNIX socket transport module. [ 1.013817] RPC: Registered udp transport module. [ 1.015410] RPC: Registered tcp transport module. [ 1.018860] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.021166] NET: Registered protocol family 44 [ 1.022685] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.024579] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.027446] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.029683] PCI: CLS 0 bytes, default 64 [ 1.031481] Unpacking initramfs... [ 2.958139] debug: unmapping init [mem 0xffff8bf93cc54000-0xffff8bf93ffbffff] [ 2.962569] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.965864] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.969620] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.674760] Initialise system trusted keyrings [ 3.676561] Key type blacklist registered [ 3.678778] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.689111] zbud: loaded [ 3.693143] *** VALIDATE nfs *** [ 3.694852] *** VALIDATE nfs4 *** [ 3.696607] pstore: using deflate compression [ 3.701350] Platform Keyring initialized [ 3.849763] NET: Registered protocol family 38 [ 3.851854] Key type asymmetric registered [ 3.853580] Asymmetric key parser 'x509' registered [ 3.855559] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.858633] io scheduler mq-deadline registered [ 3.860369] io scheduler kyber registered [ 3.861798] io scheduler bfq registered [ 3.863598] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.866304] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.868513] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.871528] ACPI: Power Button [PWRF] [ 3.878598] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.889430] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.915695] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.937698] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.979851] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 4.013824] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 4.048454] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 4.053373] Non-volatile memory driver v1.3 [ 4.055084] Linux agpgart interface v0.103 [ 4.100621] virtio_blk virtio1: [vda] 150400 512-byte logical blocks (77.0 MB/73.4 MiB) [ 4.107528] vda: detected capacity change from 0 to 77004800 [ 4.134722] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.140708] vdb: detected capacity change from 0 to 1073741824 [ 4.161305] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.165630] vdc: detected capacity change from 0 to 2621440000 [ 4.186589] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.191798] vdd: detected capacity change from 0 to 2621440000 [ 4.216421] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.221131] vde: detected capacity change from 0 to 4294967296 [ 4.245445] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.250231] vdf: detected capacity change from 0 to 4294967296 [ 4.261652] libphy: Fixed MDIO Bus: probed [ 4.276271] usbcore: registered new interface driver usbserial_generic [ 4.278126] usbserial: USB Serial support registered for generic [ 4.282333] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.287343] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.289069] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.291472] mousedev: PS/2 mouse device common for all mice [ 4.293844] rtc_cmos 00:05: RTC can wake from S4 [ 4.298050] rtc_cmos 00:05: registered as rtc0 [ 4.300356] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.303813] intel_pstate: CPU model not supported [ 4.307777] hid: raw HID events driver (C) Jiri Kosina [ 4.308407] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.312508] usbcore: registered new interface driver usbhid [ 4.312515] usbhid: USB HID core driver [ 4.312627] drop_monitor: Initializing network drop monitor service [ 4.312754] Initializing XFRM netlink socket [ 4.313467] NET: Registered protocol family 10 [ 4.314769] Segment Routing with IPv6 [ 4.314801] NET: Registered protocol family 17 [ 4.314909] mpls_gso: MPLS GSO support [ 4.330032] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.331572] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.350950] RAS: Correctable Errors collector initialized. [ 4.352767] AVX version of gcm_enc/dec engaged. [ 4.354461] AES CTR mode by8 optimization enabled [ 4.438253] sched_clock: Marking stable (4438172572, 0)->(5612710155, -1174537583) [ 4.442548] registered taskstats version 1 [ 4.444773] Loading compiled-in X.509 certificates [ 4.447671] zswap: loaded using pool lzo/zbud [ 4.484262] Key type big_key registered [ 4.502418] Key type encrypted registered [ 4.504407] ima: No TPM chip found, activating TPM-bypass! [ 4.510268] ima: Allocated hash algorithm: sha1 [ 4.512615] ima: No architecture policies found [ 4.515844] evm: Initialising EVM extended attributes: [ 4.518259] evm: security.selinux [ 4.519951] evm: security.ima [ 4.521573] evm: security.capability [ 4.523415] evm: HMAC attrs: 0x1 [ 4.526371] rtc_cmos 00:05: setting system clock to 2026-09-21 13:03:42 UTC (1789995822) [ 4.535983] debug: unmapping init [mem 0xffffffffba203000-0xffffffffba3fffff] [ 4.539848] debug: unmapping init [mem 0xffffffffb8f82000-0xffffffffb9258fff] [ 4.548343] Write protecting the kernel read-only data: 28672k [ 4.552845] debug: unmapping init [mem 0xffffffffb7603000-0xffffffffb77fffff] [ 4.556384] debug: unmapping init [mem 0xffffffffb7f14000-0xffffffffb7ffffff] [ 4.596740] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.607766] systemd[1]: Detected virtualization kvm. [ 4.610229] systemd[1]: Detected architecture x86-64. [ 4.612526] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.641793] systemd[1]: No hostname configured. [ 4.643780] systemd[1]: Set hostname to . [ 4.646230] random: systemd: uninitialized urandom read (16 bytes read) [ 4.648837] systemd[1]: Initializing machine ID from random generator. [ 4.739845] random: ln: uninitialized urandom read (6 bytes read) [ 4.928786] random: systemd: uninitialized urandom read (16 bytes read) [ 4.932308] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 4.942827] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 4.948231] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Timers. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. Starting Apply Kernel Variables... [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Reached target Paths. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 6.046782] device-mapper: uevent: version 1.0.3 [ 6.049513] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 7.164096] virtio_net virtio0 ens2: renamed from eth0 [ 7.204220] random: fast init done [ 7.306165] scsi host0: ata_piix [ 7.380694] scsi host1: ata_piix [ 7.382454] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 7.384525] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 12.079747] random: crng init done [ 12.086976] random: 7 urandom warning(s) missed due to ratelimiting [ 12.987884] dracut-initqueue[589]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 14.464617] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 16.551919] printk: systemd: 25 output lines suppressed due to ratelimiting [ 17.419166] SELinux: Disabled at runtime. [ 17.540378] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 17.558600] systemd[1]: Detected virtualization kvm. [ 17.562506] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 18.929828] systemd[1]: initrd-switch-root.service: Succeeded. [ 18.937317] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 18.945047] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 18.950612] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 18.959679] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 18.976191] systemd[1]: Starting Journal Service... Starting Journal Service... [ 18.983117] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Stopped target Switch Root. [ OK ] Created slice system-getty.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on initctl Compatibility Named Pipe. Mounting Huge Pages File System... Starting Apply Kernel Variables... Mounting POSIX Message Queue File System... Activating swap /dev/disk/by-label/SWAP... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on Process Core Du[ 19.220535] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS mp Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Starting udev Coldplug all Devices... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Stopped target Initrd File Systems. Starting Create list of required st…ce nodes for the current kernel... Mounting Kernel Debug File System... [ OK ] Stopped target Initrd Root File System. [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started udev Coldplug all Devices. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 20.017496] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 20.658204] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 20.794378] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 21.090875] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 21.138801] EDAC sbridge: Ver: 1.1.2 [ 24.451724] Key type dns_resolver registered [ 25.110153] NFS: Registering the id_resolver key type [ 25.115382] Key type id_resolver registered [ 25.119052] Key type id_legacy registered [* ] A start job is running for Configur…-only root support (6s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started OpenSSH server daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg428-server login: [ 50.556154] libcfs: loading out-of-tree module taints kernel. [ 50.574214] Key type ._llcrypt registered [ 50.575858] Key type .llcrypt registered [ 50.617537] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_hostid [ 58.276180] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 58.840496] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 58.846289] alg: No test for adler32 (adler32-zlib) [ 59.812018] Lustre: Lustre: Build Version: 2.17.58_63_g3df9749 [ 60.130307] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 61.735187] Key type lgssc registered [ 62.342220] Lustre: Echo OBD driver; http://www.lustre.org/ [ 68.708801] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 83.534983] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 88.257615] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 88.266875] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 89.356180] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 89.377218] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 89.416652] Lustre: lustre-MDT0000: new disk, initializing [ 89.446494] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 89.456365] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 90.841708] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 95.881919] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 95.919556] Lustre: 6477:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 95.930274] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 95.932112] Lustre: Skipped 1 previous similar message [ 95.966234] Lustre: lustre-MDT0001: new disk, initializing [ 95.986721] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 95.995466] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 96.000567] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 97.345400] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 99.585982] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 102.648632] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 102.733596] Lustre: lustre-OST0000: new disk, initializing [ 102.735452] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 102.738554] Lustre: 8415:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 102.757511] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 104.236296] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 104.239141] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 104.263100] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 104.700315] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 110.399188] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 110.453610] Lustre: lustre-OST0001: new disk, initializing [ 110.455856] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 110.459206] Lustre: 9488:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 110.486548] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 110.966248] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 110.969837] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 110.984183] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 112.487880] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 118.143479] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 123.767190] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 125.883455] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing check_logdir /tmp/testlogs/ [ 127.397504] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing yml_node [ 128.912334] Lustre: DEBUG MARKER: Client: 2.17.58.63 [ 129.956387] Lustre: DEBUG MARKER: MDS: 2.17.58.63 [ 131.017960] Lustre: DEBUG MARKER: OSS: 2.17.58.63 [ 131.721183] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Sep 21 09:05:47 EDT 2026 [ 145.165262] Lustre: DEBUG MARKER: only running tests: 33-75 [ 147.347913] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 149.925483] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 172.514912] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 172.516790] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 172.554356] Lustre: Skipped 1 previous similar message [ 172.562080] Lustre: Skipped 1 previous similar message [ 177.765267] Lustre: server umount lustre-MDT0000 complete [ 180.572799] hrtimer: interrupt took 4425314 ns [ 182.755949] LustreError: 7684:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 182.795084] LustreError: 7684:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [ 185.826726] LustreError: 6469:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789996003 with bad export cookie 2837126093027033445 [ 185.842203] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 187.875247] LustreError: 6485:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 187.875860] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 187.891504] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 187.891515] Lustre: Skipped 1 previous similar message [ 187.924722] LustreError: 6485:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 187.940891] Lustre: Skipped 3 previous similar messages [ 190.947880] LustreError: 7684:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 190.949316] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 190.969077] LustreError: 7684:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 190.982650] Lustre: Skipped 2 previous similar messages [ 192.571390] Lustre: server umount lustre-MDT0001 complete [ 198.272961] Lustre: server umount lustre-OST0000 complete [ 204.490514] Lustre: server umount lustre-OST0001 complete [ 213.931900] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_hostid [ 225.779918] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 259.843636] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 260.060278] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 260.113963] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 260.169416] Lustre: lustre-MDT0000: new disk, initializing [ 260.251326] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 260.272902] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 263.777668] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 272.158128] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 272.269754] Lustre: 15086:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 272.341249] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 272.351380] Lustre: Skipped 1 previous similar message [ 272.438895] Lustre: lustre-MDT0001: new disk, initializing [ 272.535021] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 272.579633] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 272.595678] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 276.365245] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 284.508444] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 289.575719] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 295.890914] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 296.217392] Lustre: lustre-OST0000: new disk, initializing [ 296.222859] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 296.231740] Lustre: 17054:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 296.316842] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 297.768920] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 297.810133] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 297.960894] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 302.645560] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 311.431731] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 316.264448] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 316.461720] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 320.929779] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 321.179655] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 323.555188] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 323.556687] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 323.564703] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 323.577525] Lustre: Skipped 1 previous similar message [ 328.675313] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 328.680967] Lustre: Skipped 1 previous similar message [ 337.375496] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 337.625412] Lustre: server umount lustre-OST0000 complete [ 344.043540] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 344.082275] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 344.086188] Lustre: Skipped 1 previous similar message [ 344.106218] Lustre: Skipped 4 previous similar messages [ 349.116481] Lustre: server umount lustre-MDT0000 complete [ 349.157940] LustreError: 17062:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 349.185179] LustreError: 17062:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 354.273108] LustreError: 15094:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 354.297807] LustreError: 15094:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 354.802875] LustreError: 15079:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789996172 with bad export cookie 2837126093027034957 [ 354.805653] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 354.825072] LustreError: 15079:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 355.421040] Lustre: server umount lustre-MDT0001 complete [ 359.982444] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 09:09:35 (1789996175) [ 363.273827] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 365.655747] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 09:09:40 (1789996180) [ 372.802995] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 373.370836] LustreError: 19467:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 373.501404] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 378.030464] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 386.357713] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 391.314291] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 398.258794] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 403.115355] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 409.445541] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 416.616318] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 424.909331] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 432.608698] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 432.618037] LustreError: Skipped 1 previous similar message [ 432.626358] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 432.647796] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 433.636732] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 437.967290] Lustre: server umount lustre-OST0000 complete [ 443.361087] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 443.371834] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 443.386722] Lustre: Skipped 1 previous similar message [ 456.159172] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 456.401727] Lustre: server umount lustre-MDT0000 complete [ 458.214875] LustreError: 20500:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 458.225751] LustreError: 20500:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 459.807957] LustreError: 19449:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789996277 with bad export cookie 2837126093027035846 [ 459.822878] LustreError: 19449:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 459.828712] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 460.089733] Lustre: server umount lustre-MDT0001 complete [ 469.528848] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 471.960734] Key type lgssc unregistered [ 472.224220] LNet: 22998:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 472.240330] LNetError: 22998:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 472.252282] LNet: Removed LNI 192.168.204.128@tcp [ 473.082176] Key type .llcrypt unregistered [ 473.085063] Key type ._llcrypt unregistered [ 498.546935] Key type ._llcrypt registered [ 498.555415] Key type .llcrypt registered [ 498.719183] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 499.806771] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 499.967647] alg: No test for adler32 (adler32-zlib) [ 500.957124] Lustre: Lustre: Build Version: 2.17.58_63_g3df9749 [ 501.197280] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 502.856490] Key type lgssc registered [ 504.111658] Lustre: Echo OBD driver; http://www.lustre.org/ [ 511.162216] loop: module loaded [ 511.232754] loop0: detected capacity change from 0 to 204800000 [ 511.452293] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 511.466559] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 511.893681] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 516.813567] loop0: detected capacity change from 0 to 204800000 [ 517.015675] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 517.043713] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 517.079679] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 521.797240] loop0: detected capacity change from 0 to 204800000 [ 526.338387] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 526.358349] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 527.605163] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 527.654806] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 527.720086] Lustre: lustre-MDT0000: new disk, initializing [ 527.822820] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 527.853728] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 533.262571] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 541.036408] loop1: detected capacity change from 0 to 204800000 [ 546.318801] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 546.620094] Lustre: lustre-OST07c6: new disk, initializing [ 546.631700] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 546.645967] Lustre: Skipped 1 previous similar message [ 546.658555] Lustre: 26196:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 546.799845] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 555.635076] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 556.707633] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 556.713848] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 556.753704] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 562.433418] Lustre: 25183:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 562.439712] Lustre: 25183:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 562.444725] Lustre: 25183:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 562.450417] Lustre: 25183:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 562.456143] Lustre: 25183:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 562.460571] Lustre: 25183:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 580.584890] Lustre: 25184:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 580.596454] Lustre: 25184:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 580.604855] Lustre: 25184:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 580.611728] Lustre: 25184:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 580.618727] Lustre: 25184:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 580.631415] Lustre: 25184:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 585.877335] Lustre: 26204:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 585.892568] Lustre: 26204:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 585.902615] Lustre: 26204:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 585.911617] Lustre: 26204:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 585.919901] Lustre: 26204:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 585.930067] Lustre: 26204:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 585.935203] Lustre: 26204:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 585.942294] Lustre: 26204:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 585.948406] Lustre: 26204:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 585.957423] Lustre: 26204:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 585.964259] Lustre: 26204:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 585.980989] Lustre: 26204:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 590.617542] Lustre: 25183:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 590.625047] Lustre: 25183:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 590.633291] Lustre: 25183:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 590.643404] Lustre: 25183:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 590.651647] Lustre: 25183:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 590.657668] Lustre: 25183:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 590.673073] Lustre: 25183:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 590.693271] Lustre: 25183:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 590.712677] Lustre: 25183:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 590.724673] Lustre: 25183:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 590.742542] Lustre: 25183:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 590.761384] Lustre: 25183:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 598.712296] Lustre: 25185:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 598.736988] Lustre: 25185:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 3 previous similar messages [ 598.746571] Lustre: 25185:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 598.752301] Lustre: 25185:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 598.760161] Lustre: 25185:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 598.766521] Lustre: 25185:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 598.773358] Lustre: 25185:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 598.781478] Lustre: 25185:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 598.789851] Lustre: 25185:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 598.797989] Lustre: 25185:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 598.804440] Lustre: 25185:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 598.810732] Lustre: 25185:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 609.934774] Lustre: 26204:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 609.944757] Lustre: 26204:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 5 previous similar messages [ 609.952299] Lustre: 26204:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 609.960258] Lustre: 26204:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 609.968277] Lustre: 26204:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 609.972966] Lustre: 26204:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 609.986248] Lustre: 26204:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 609.993666] Lustre: 26204:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 610.000302] Lustre: 26204:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 610.011069] Lustre: 26204:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 610.022240] Lustre: 26204:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 610.030506] Lustre: 26204:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 653.804814] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 653.822343] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 657.945276] Lustre: server umount lustre-OST07c6 complete [ 666.814547] Lustre: server umount lustre-MDT0000 complete [ 680.400213] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 683.153825] Key type lgssc unregistered [ 683.414562] LNet: 28753:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 683.419398] LNetError: 28753:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 683.433286] LNet: Removed LNI 192.168.204.128@tcp [ 684.180155] Key type .llcrypt unregistered [ 684.190586] Key type ._llcrypt unregistered [ 699.744975] Key type ._llcrypt registered [ 699.747648] Key type .llcrypt registered [ 699.892772] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 700.837601] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 700.971889] alg: No test for adler32 (adler32-zlib) [ 702.023087] Lustre: Lustre: Build Version: 2.17.58_63_g3df9749 [ 702.233200] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 703.911194] Key type lgssc registered [ 704.782703] Lustre: Echo OBD driver; http://www.lustre.org/ [ 711.340347] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 711.352813] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 712.656155] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 715.994232] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 722.914553] LustreError: 30020:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 722.933759] LustreError: 30020:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 723.446390] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 723.837614] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 727.854514] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 734.642564] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 740.015178] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 746.075475] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 746.423051] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 751.567850] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 757.688411] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 757.802409] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 773.090726] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 773.092049] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 773.096610] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 773.111428] Lustre: Skipped 1 previous similar message [ 777.531280] Lustre: server umount lustre-OST0000 complete [ 783.329382] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 783.332289] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 783.347914] Lustre: Skipped 1 previous similar message [ 783.367415] Lustre: Skipped 2 previous similar messages [ 787.471575] Lustre: server umount lustre-MDT0000 complete [ 788.448253] LustreError: 30749:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 788.476855] LustreError: 30749:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 790.905470] LustreError: 30000:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789996608 with bad export cookie 5557383159995926387 [ 790.923119] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 790.926590] LustreError: 30000:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 791.319204] Lustre: server umount lustre-MDT0001 complete [ 802.207848] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 805.143768] Key type lgssc unregistered [ 805.560592] LNet: 33755:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 805.576843] LNetError: 33755:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 805.593766] LNet: Removed LNI 192.168.204.128@tcp [ 806.397735] Key type .llcrypt unregistered [ 806.401534] Key type ._llcrypt unregistered [ 826.417328] Key type ._llcrypt registered [ 826.420630] Key type .llcrypt registered [ 826.501668] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 827.802587] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 827.822111] alg: No test for adler32 (adler32-zlib) [ 828.870302] Lustre: Lustre: Build Version: 2.17.58_63_g3df9749 [ 829.125237] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 830.823238] Key type lgssc registered [ 831.969747] Lustre: Echo OBD driver; http://www.lustre.org/ [ 838.244602] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 838.256346] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 839.667715] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 843.501488] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 849.889844] LustreError: 35042:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 849.904308] LustreError: 35042:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 851.164113] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 851.493814] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 855.308672] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 861.823943] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 865.886180] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 870.831866] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 871.129214] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 875.667344] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 882.220620] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 884.480439] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 896.992343] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 897.006516] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 897.020129] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 900.067626] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 900.071045] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 900.104250] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 901.466130] Lustre: server umount lustre-OST0000 complete [ 905.187214] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 905.200822] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 905.203799] Lustre: Skipped 1 previous similar message [ 905.215986] Lustre: Skipped 2 previous similar messages [ 910.306734] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 910.314969] Lustre: Skipped 1 previous similar message [ 911.047880] Lustre: server umount lustre-MDT0000 complete [ 914.140063] LustreError: 35022:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789996732 with bad export cookie 1138109590945460617 [ 914.154139] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 914.159786] LustreError: 35022:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 914.431928] Lustre: server umount lustre-MDT0001 complete [ 923.749049] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 926.014716] Key type lgssc unregistered [ 926.220578] LNet: 38597:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 926.225479] LNetError: 38597:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 926.243553] LNet: Removed LNI 192.168.204.128@tcp [ 927.070988] Key type .llcrypt unregistered [ 927.073708] Key type ._llcrypt unregistered [ 948.545595] Key type ._llcrypt registered [ 948.554646] Key type .llcrypt registered [ 948.665125] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 949.696764] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 949.819631] alg: No test for adler32 (adler32-zlib) [ 950.894517] Lustre: Lustre: Build Version: 2.17.58_63_g3df9749 [ 951.174615] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 952.871295] Key type lgssc registered [ 954.158157] Lustre: Echo OBD driver; http://www.lustre.org/ [ 961.468923] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 961.485092] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 962.872852] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 967.221228] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 972.271648] LustreError: 40027:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 972.295251] LustreError: 40027:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 975.326266] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 975.694717] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 979.769380] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 986.605604] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 992.074064] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 998.432812] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 998.809525] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1004.944499] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 1008.074140] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1011.897572] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1017.988969] Lustre: server umount lustre-MDT0000 complete [ 1018.338886] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1018.350245] LustreError: 40023:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1018.363725] Lustre: Skipped 2 previous similar messages [ 1018.397287] LustreError: 40023:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1021.701705] LustreError: 40010:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789996839 with bad export cookie 16056052739470005 [ 1021.715165] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1021.722441] LustreError: 40010:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1023.456865] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1023.461496] LustreError: 40022:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1023.477736] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1023.498918] LustreError: 40022:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1026.466466] LustreError: 40022:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1026.466760] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1026.487341] LustreError: 40022:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1026.502439] Lustre: Skipped 1 previous similar message [ 1028.734212] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.28@tcp (stopping) [ 1031.587593] LustreError: 41081:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1031.588455] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1031.598807] LustreError: 41081:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1036.255142] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1036.406966] Lustre: server umount lustre-MDT0001 complete [ 1058.050419] Lustre: server umount lustre-OST0000 complete [ 1076.300508] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 1079.084936] Key type lgssc unregistered [ 1079.377799] LNet: 43888:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1079.386456] LNetError: 43888:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1079.404707] LNet: Removed LNI 192.168.204.128@tcp [ 1080.182946] Key type .llcrypt unregistered [ 1080.185592] Key type ._llcrypt unregistered [ 1103.722320] Key type ._llcrypt registered [ 1103.726232] Key type .llcrypt registered [ 1103.847464] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 1105.416670] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1105.447369] alg: No test for adler32 (adler32-zlib) [ 1106.592189] Lustre: Lustre: Build Version: 2.17.58_63_g3df9749 [ 1106.876835] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 1108.583266] Key type lgssc registered [ 1109.757161] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1117.342910] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1117.382752] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1119.014837] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1123.327266] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 1128.419262] LustreError: 45320:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1128.437595] LustreError: 45320:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1132.486695] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1133.035136] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1137.129682] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 1144.284301] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1150.304489] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1156.344213] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1156.617221] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1163.368882] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 1163.816350] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1171.070357] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1177.731985] Lustre: lustre-OST0000: Not available for connect from 192.168.204.28@tcp (stopping) [ 1178.083646] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1178.089901] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1179.112650] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1179.118388] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1179.120833] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1179.139237] Lustre: Skipped 2 previous similar messages [ 1182.833567] Lustre: lustre-OST0000: Not available for connect from 192.168.204.28@tcp (stopping) [ 1187.956820] Lustre: lustre-OST0000: Not available for connect from 192.168.204.28@tcp (stopping) [ 1187.962979] Lustre: Skipped 2 previous similar messages [ 1190.367227] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1190.522703] Lustre: server umount lustre-OST0000 complete [ 1209.835197] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1209.838263] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1209.838498] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1209.846361] Lustre: Skipped 1 previous similar message [ 1209.853068] Lustre: Skipped 2 previous similar messages [ 1214.474934] Lustre: server umount lustre-MDT0000 complete [ 1214.946360] LustreError: 47163:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1214.972565] LustreError: 47163:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1218.225267] LustreError: 45301:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789997036 with bad export cookie 16301810362361846607 [ 1218.245434] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1218.249413] LustreError: 45301:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1224.722956] Lustre: server umount lustre-MDT0001 complete [ 1234.610749] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 1237.199863] Key type lgssc unregistered [ 1237.454626] LNet: 49030:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1237.462068] LNetError: 49030:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1237.483123] LNet: Removed LNI 192.168.204.128@tcp [ 1238.233644] Key type .llcrypt unregistered [ 1238.240970] Key type ._llcrypt unregistered [ 1259.773426] Key type ._llcrypt registered [ 1259.776188] Key type .llcrypt registered [ 1259.843582] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 1260.450426] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1260.513087] alg: No test for adler32 (adler32-zlib) [ 1261.635610] Lustre: Lustre: Build Version: 2.17.58_63_g3df9749 [ 1261.845152] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 1263.487420] Key type lgssc registered [ 1264.834498] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1273.094121] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1273.105888] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1274.568033] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1279.835467] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 1284.068838] LustreError: 50461:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1284.087982] LustreError: 50461:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1289.059087] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1289.186877] LustreError: 50462:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1289.489388] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1293.541688] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 1300.069469] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1304.924896] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1310.271104] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1310.545171] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1315.442116] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 1319.811260] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1321.838134] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1326.182539] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1328.801331] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1340.546572] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-09-21 9h25m55s [ 1341.935423] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1344.157911] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.28@tcp (stopping) [ 1345.203989] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.28@tcp (stopping) [ 1345.510563] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1346.605171] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1346.621772] Lustre: Skipped 5 previous similar messages [ 1350.166918] Lustre: server umount lustre-MDT0000 complete [ 1350.264356] LustreError: 52263:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1352.681449] LustreError: 53357:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1352.704600] LustreError: 53357:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1353.291331] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1358.819795] LustreError: 53357:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1358.833641] LustreError: 53357:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 1358.849673] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1358.953372] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1359.199173] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1359.268325] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1360.236278] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1362.864128] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 1365.339967] Lustre: DEBUG MARKER: Wait for df (36986) ... [ 1366.522852] Lustre: DEBUG MARKER: done [ 1369.571875] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1369.582204] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1369.597434] Lustre: Skipped 2 previous similar messages [ 1369.605056] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1370.600453] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1375.540475] Lustre: server umount lustre-OST0000 complete [ 1380.833580] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1380.834246] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1380.835252] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1380.835259] Lustre: Skipped 2 previous similar messages [ 1380.842409] Lustre: Skipped 1 previous similar message [ 1384.521632] Lustre: server umount lustre-MDT0000 complete [ 1387.587198] LustreError: 50441:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789997205 with bad export cookie 10123651115685033193 [ 1387.590756] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1387.594656] LustreError: 50441:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1387.875700] Lustre: server umount lustre-MDT0001 complete [ 1396.010268] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 1398.468223] Key type lgssc unregistered [ 1398.717383] LNet: 55381:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1398.740387] LNetError: 55381:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1398.764752] LNet: Removed LNI 192.168.204.128@tcp [ 1399.692324] Key type .llcrypt unregistered [ 1399.698240] Key type ._llcrypt unregistered [ 1431.533712] Key type ._llcrypt registered [ 1431.537217] Key type .llcrypt registered [ 1431.634561] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 1432.263218] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1432.311220] alg: No test for adler32 (adler32-zlib) [ 1433.265600] Lustre: Lustre: Build Version: 2.17.58_63_g3df9749 [ 1433.459472] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 1435.111881] Key type lgssc registered [ 1435.902797] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1441.519756] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1441.546026] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1442.787231] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1442.810253] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1442.937740] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1445.801819] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 1452.408442] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1452.471365] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1452.492118] Lustre: 57583:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1452.697492] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1456.602969] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 1462.201095] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1466.651618] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1472.088899] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1472.244668] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1472.414565] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1477.384286] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 1478.787730] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1483.053081] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1489.242842] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-09-21 9h28m25s [ 1490.467718] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1495.995984] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1497.683313] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1497.731296] LustreError: 57589:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1518.225962] Lustre: lustre-MDT0000: Client 32c64990-31c7-40d6-b189-f91357147158 (at 192.168.204.28@tcp) reconnecting [ 1537.831141] LustreError: 57589:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1539.266179] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1540.881314] Lustre: DEBUG MARKER: done [ 1545.184362] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1545.192361] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1545.202761] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1545.698477] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1545.701755] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1545.709620] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1550.428924] Lustre: server umount lustre-OST0000 complete [ 1555.424413] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1555.430135] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1555.440202] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1559.219783] Lustre: server umount lustre-MDT0000 complete [ 1561.942301] LustreError: 57573:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789997379 with bad export cookie 11690353550594438165 [ 1561.946502] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1561.948076] LustreError: 57573:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1562.189384] Lustre: server umount lustre-MDT0001 complete [ 1570.432480] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 1572.332344] Key type lgssc unregistered [ 1572.555410] LNet: 61910:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1572.562903] LNetError: 61910:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1572.576913] LNet: Removed LNI 192.168.204.128@tcp [ 1573.084666] Key type .llcrypt unregistered [ 1573.087324] Key type ._llcrypt unregistered [ 1602.018486] Key type ._llcrypt registered [ 1602.020691] Key type .llcrypt registered [ 1602.099423] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 1602.571942] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1602.601780] alg: No test for adler32 (adler32-zlib) [ 1603.529915] Lustre: Lustre: Build Version: 2.17.58_63_g3df9749 [ 1603.676980] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 1605.295212] Key type lgssc registered [ 1606.079359] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1608.396550] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1608.415772] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1609.643871] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1609.662212] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1609.784682] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1610.483724] Lustre: Failing over lustre-MDT0000 [ 1610.713654] Lustre: server umount lustre-MDT0000 complete [ 1614.759335] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 09:30:30 (1789997430) [ 1618.557087] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1618.813365] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1621.572386] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 1626.331248] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1626.388265] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1626.409387] Lustre: 64701:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1626.558191] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1628.808192] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 1632.595543] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1635.849698] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1639.695830] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1639.839123] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1639.970858] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1643.176617] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 1646.655766] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1647.515439] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1652.386084] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1656.801648] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1656.804028] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1656.809809] Lustre: Skipped 1 previous similar message [ 1661.831978] Lustre: server umount lustre-MDT0000 complete [ 1663.843064] LustreError: 64691:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789997481 with bad export cookie 14158156855120371302 [ 1663.847128] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1663.848775] LustreError: 64691:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1664.018988] Lustre: server umount lustre-MDT0001 complete [ 1665.872176] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1668.373401] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1673.267676] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1683.423175] Lustre: 63615:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789997485/real 1789997485] req@000000005cb8e4b8 x1876948335757184/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789997501 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1683.444690] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1683.451885] Lustre: Skipped 1 previous similar message [ 1684.511234] Lustre: 63615:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789997486/real 1789997486] req@00000000982e80b0 x1876948335757312/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789997502 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1688.543161] Lustre: 63617:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789997490/real 1789997490] req@000000005bf31562 x1876948335757440/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789997506 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1688.608081] LustreError: 63613:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000ce712146 x1876948335758336/t0(0) o250->MGC192.168.204.128@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1688.789743] LustreError: 66547:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1688.798567] LustreError: 66547:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 1688.858537] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1688.912545] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1690.977136] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 1692.897742] LustreError: 66548:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1693.922869] LustreError: 68470:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1693.933978] LustreError: 68470:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1694.944092] Lustre: 63615:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789997496/real 1789997496] req@000000006de3c8d3 x1876948335757824/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789997512 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1694.957633] Lustre: 63615:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1696.039432] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1699.038841] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 1699.360409] Lustre: 63614:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789997501/real 1789997501] req@00000000180fea2b x1876948335757952/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789997517 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1701.352496] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1704.170493] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1707.872318] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1712.930813] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1712.933054] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1712.940194] Lustre: Skipped 3 previous similar messages [ 1712.958441] Lustre: Skipped 1 previous similar message [ 1718.088559] Lustre: server umount lustre-MDT0000 complete [ 1718.243123] LustreError: 68471:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1718.254974] LustreError: 68471:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1720.559167] LustreError: 66555:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789997538 with bad export cookie 14158156855120373577 [ 1720.562894] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1720.573134] LustreError: 66555:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1720.788464] Lustre: server umount lustre-MDT0001 complete [ 1724.691236] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1730.212130] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1730.531394] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xc47be0a587b62ca5 [ 1730.536256] Lustre: MGC192.168.204.128@tcp: Connection restored to 0@lo (at 0@lo) [ 1730.704548] LustreError: 69203:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1730.763135] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1730.767209] Lustre: Skipped 1 previous similar message [ 1730.796105] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 1733.683344] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 1736.162660] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1739.629584] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1739.679312] Lustre: 63614:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789997541/real 1789997541] req@000000009f895172 x1876948335780480/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789997557 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1742.676935] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 1744.876028] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1747.505142] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1748.970468] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1751.539916] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1755.104779] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1755.110409] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1755.119532] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1755.123611] Lustre: Skipped 1 previous similar message [ 1759.203124] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1761.056746] Lustre: server umount lustre-MDT0000 complete [ 1763.124692] LustreError: 64691:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789997581 with bad export cookie 14158156855120374949 [ 1763.129597] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1763.138257] LustreError: 64691:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1763.281491] Lustre: server umount lustre-MDT0001 complete [ 1767.115690] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1769.261288] Lustre: DEBUG MARKER: files compared the same [ 1781.027144] Lustre: server umount lustre-OST0000 complete [ 1791.595693] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 1793.493587] Key type lgssc unregistered [ 1793.737538] LNet: 74421:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1793.744830] LNetError: 74421:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1793.761900] LNet: Removed LNI 192.168.204.128@tcp [ 1794.285740] Key type .llcrypt unregistered [ 1794.287272] Key type ._llcrypt unregistered [ 1810.977727] Key type ._llcrypt registered [ 1810.979491] Key type .llcrypt registered [ 1811.064857] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 1811.711233] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1811.749958] alg: No test for adler32 (adler32-zlib) [ 1812.715544] Lustre: Lustre: Build Version: 2.17.58_63_g3df9749 [ 1812.922473] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 1814.551214] Key type lgssc registered [ 1815.344219] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1820.631501] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1820.651896] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1821.981329] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1825.052406] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug +malloc all [ 1831.399235] LustreError: 75833:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1831.408210] LustreError: 75833:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1831.441825] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1831.624647] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1834.771273] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug +malloc all [ 1839.539675] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1843.160171] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1848.358889] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1848.584822] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1853.063411] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug +malloc all [ 1860.151377] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1867.052602] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 1882.593043] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1882.610189] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1882.635796] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1887.713711] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1887.722249] Lustre: Skipped 1 previous similar message [ 1888.154577] Lustre: server umount lustre-OST0000 complete [ 1892.832719] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1892.840728] Lustre: Skipped 1 previous similar message [ 1892.843981] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1892.847551] Lustre: Skipped 1 previous similar message [ 1897.414973] Lustre: server umount lustre-MDT0000 complete [ 1897.953281] LustreError: 77500:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1897.966252] LustreError: 77500:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1900.474966] LustreError: 75813:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789997718 with bad export cookie 2072171456233587175 [ 1900.477967] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1900.481860] LustreError: 75813:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1900.737267] Lustre: server umount lustre-MDT0001 complete [ 1908.629080] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 1910.627486] Key type lgssc unregistered [ 1910.824066] LNet: 79383:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1910.828356] LNetError: 79383:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1910.843224] LNet: Removed LNI 192.168.204.128@tcp [ 1911.397557] Key type .llcrypt unregistered [ 1911.400807] Key type ._llcrypt unregistered [ 1931.579588] Key type ._llcrypt registered [ 1931.581503] Key type .llcrypt registered [ 1931.707773] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 1933.133068] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1933.260405] alg: No test for adler32 (adler32-zlib) [ 1934.494730] Lustre: Lustre: Build Version: 2.17.58_63_g3df9749 [ 1934.914979] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 1936.664603] Key type lgssc registered [ 1938.675785] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1946.996726] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1947.024267] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1962.591445] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1967.711316] LustreError: 80787:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.128@tcp: failed processing log, type 4: rc = -110 [ 1994.399325] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1999.056174] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 2003.801229] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2008.533330] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2008.641454] Lustre: *** cfs_fail_loc=706, val=0*** [ 2008.892608] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2008.937051] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2011.656341] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 2015.203102] LustreError: 81873:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2015.215237] LustreError: 81873:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 2018.410882] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2018.661129] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2021.705438] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 2027.734786] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2032.552032] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2034.656566] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2034.661212] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2034.679546] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2039.269822] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2039.274302] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2039.274807] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2040.744299] Lustre: server umount lustre-OST0000 complete [ 2044.388486] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2044.393429] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2044.400954] Lustre: Skipped 1 previous similar message [ 2044.402479] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2044.402485] Lustre: Skipped 1 previous similar message [ 2049.506819] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2049.517246] Lustre: Skipped 2 previous similar messages [ 2050.198904] Lustre: server umount lustre-MDT0000 complete [ 2053.429151] LustreError: 80793:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789997871 with bad export cookie 14061067726546326300 [ 2053.445404] LustreError: 80793:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2053.445761] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2053.751220] Lustre: server umount lustre-MDT0001 complete [ 2064.025079] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 2066.675877] Key type lgssc unregistered [ 2067.064875] LNet: 84397:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2067.071965] LNetError: 84397:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2067.085953] LNet: Removed LNI 192.168.204.128@tcp [ 2067.959301] Key type .llcrypt unregistered [ 2067.961345] Key type ._llcrypt unregistered [ 2091.363726] Key type ._llcrypt registered [ 2091.367109] Key type .llcrypt registered [ 2091.495865] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 2092.593727] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2092.619256] alg: No test for adler32 (adler32-zlib) [ 2093.693591] Lustre: Lustre: Build Version: 2.17.58_63_g3df9749 [ 2093.913399] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 2095.559195] Key type lgssc registered [ 2096.613338] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2103.954258] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2103.966568] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2109.465526] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 2116.550579] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2117.079579] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2121.253404] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 2122.209305] LustreError: 86527:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2122.233245] LustreError: 86527:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2126.304971] LustreError: 86526:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2126.335752] LustreError: 86526:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2129.062142] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2129.306706] LustreError: 86527:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2129.342409] LustreError: 86527:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2129.343615] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2133.968880] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 2134.497911] LustreError: 86531:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2134.510601] LustreError: 86531:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2139.620271] LustreError: 86530:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2139.640597] LustreError: 86530:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2140.120205] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2140.139509] Lustre: MGS: Client edd25ed0-fbf3-4e44-a688-1f39d0ba562f (at 0@lo) reconnecting [ 2140.158434] LustreError: 86541:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2140.356892] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2140.420230] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2143.599115] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 2156.001527] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2156.005757] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2156.023743] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2157.026926] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2157.027680] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2157.030658] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2161.106557] Lustre: server umount lustre-OST0000 complete [ 2167.266174] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2167.283714] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2167.287420] Lustre: Skipped 2 previous similar messages [ 2170.247857] Lustre: server umount lustre-MDT0000 complete [ 2172.385389] LustreError: 86530:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2172.400621] LustreError: 86530:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2173.214034] Lustre: server umount lustre-MDT0001 complete [ 2175.765882] Lustre: server umount MGS complete [ 2183.636818] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 09:39:59 (1789997999) [ 2202.273541] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_hostid [ 2209.095742] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 2235.244683] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2239.144766] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 2247.118683] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2247.345597] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2247.400067] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2247.460370] Lustre: lustre-MDT0001: new disk, initializing [ 2247.522130] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2247.543261] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2247.550753] Lustre: 93688:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2247.562043] LustreError: 93688:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2251.003473] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 2259.014645] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2259.183546] Lustre: lustre-OST0000: new disk, initializing [ 2259.186948] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2259.192553] Lustre: 94453:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2259.239097] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2260.969115] Lustre: 94496:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000a87c12de x1876948849563392/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1789998094 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2264.629311] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 2272.084669] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2277.011906] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2277.023416] Lustre: MGS: Client 8f791fd7-8e67-4d53-a064-36c2f2e712f9 (at 0@lo) reconnecting [ 2277.058217] Lustre: 92957:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2277.071701] LustreError: 93674:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2277.119936] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2277.200682] Lustre: lustre-MDT0000: new disk, initializing [ 2277.283910] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2277.316699] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2277.334674] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2278.370133] Lustre: 95522:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000b9ba6325 x1876948849570176/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1789998112 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2280.678298] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 2283.491652] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2287.535148] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 2287.552291] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 2287.560819] Lustre: Skipped 1 previous similar message [ 2287.596725] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 2308.065166] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2308.071756] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2308.082166] Lustre: Skipped 1 previous similar message [ 2308.086691] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2316.259603] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2316.267283] Lustre: Skipped 4 previous similar messages [ 2322.399168] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2322.532065] Lustre: server umount lustre-OST0000 complete [ 2326.498251] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2326.499232] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2326.509245] Lustre: Skipped 1 previous similar message [ 2326.510399] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2326.517433] LustreError: Skipped 1 previous similar message [ 2326.521428] Lustre: Skipped 2 previous similar messages [ 2339.295602] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2339.364731] Lustre: server umount lustre-MDT0000 complete [ 2341.858735] LustreError: 93663:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2341.868422] LustreError: 93663:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2348.472548] Lustre: server umount lustre-MDT0001 complete [ 2351.441872] LustreError: 96667:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGS: namespace resource [0x736d61726170:0x3:0x0].0x0 (000000000734ebec) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 2353.563701] Lustre: server umount MGS complete [ 2361.709933] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 09:42:57 (1789998177) [ 2372.717973] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 2374.240536] Key type lgssc unregistered [ 2374.416943] LNet: 98020:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2374.421545] LNetError: 98020:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2374.429465] LNet: Removed LNI 192.168.204.128@tcp [ 2374.897626] Key type .llcrypt unregistered [ 2374.900337] Key type ._llcrypt unregistered [ 2384.569663] Key type ._llcrypt registered [ 2384.571904] Key type .llcrypt registered [ 2384.633826] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 2385.102877] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2385.113873] alg: No test for adler32 (adler32-zlib) [ 2386.061454] Lustre: Lustre: Build Version: 2.17.58_63_g3df9749 [ 2386.204336] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 2387.815194] Key type lgssc registered [ 2388.453445] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2393.691825] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2393.708968] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2395.007285] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2397.756289] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 2403.878561] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2404.177982] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2407.477578] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 2416.386976] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2425.093350] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2435.782784] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2436.546907] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2440.635196] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3 to 0x240000400:33) [ 2444.184799] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 2453.495887] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2465.267658] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2465.290481] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2465.292115] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2465.306113] Lustre: Skipped 1 previous similar message [ 2465.357331] Lustre: Skipped 1 previous similar message [ 2468.642323] Lustre: server umount lustre-OST0000 complete [ 2475.497711] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2475.522457] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2475.523442] Lustre: Skipped 1 previous similar message [ 2475.550396] Lustre: Skipped 1 previous similar message [ 2480.609136] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2480.623030] Lustre: Skipped 1 previous similar message [ 2481.173326] Lustre: server umount lustre-MDT0000 complete [ 2485.275045] LustreError: 99217:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789998303 with bad export cookie 17777093028659194747 [ 2485.288534] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2485.637960] Lustre: server umount lustre-MDT0001 complete [ 2495.077464] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 2497.782463] Key type lgssc unregistered [ 2498.061656] LNet: 102767:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2498.071167] LNetError: 102767:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2498.090957] LNet: Removed LNI 192.168.204.128@tcp [ 2499.012042] Key type .llcrypt unregistered [ 2499.015418] Key type ._llcrypt unregistered [ 2513.679962] Key type ._llcrypt registered [ 2513.682863] Key type .llcrypt registered [ 2513.783939] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 2514.780565] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2514.808026] alg: No test for adler32 (adler32-zlib) [ 2515.837346] Lustre: Lustre: Build Version: 2.17.58_63_g3df9749 [ 2516.155596] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 2517.911236] Key type lgssc registered [ 2519.057178] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2524.496924] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2524.500923] LustreError: 103867:0:(tgt_mount.c:2569:server_fill_super()) cfs_race id 716 sleeping [ 2524.504385] Lustre: Skipped 1 previous similar message [ 2524.512449] LustreError: 103866:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 waking [ 2524.517701] LustreError: 103867:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 awake: rc=4991 [ 2524.522226] LustreError: 103867:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 2524.530225] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2524.532521] LustreError: 103867:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2525.850163] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2532.638782] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2532.826161] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2536.572510] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 2540.596528] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing lsmod [ 2542.973951] LustreError: 105069:0:(tgt_mount.c:2569:server_fill_super()) cfs_race id 716 sleeping [ 2542.995265] LustreError: 105072:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 waking [ 2543.000483] LustreError: 105069:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 awake: rc=4980 [ 2543.007628] LustreError: 105069:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 2543.011738] LustreError: 105069:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2543.022118] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2543.357949] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2548.192220] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2548.201087] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2548.211047] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2548.713223] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2548.715904] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2548.723081] Lustre: Skipped 1 previous similar message [ 2551.877694] Lustre: server umount lustre-MDT0000 complete [ 2553.826289] LustreError: 103898:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2553.838866] LustreError: 103898:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 2554.751978] LustreError: 103880:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789998372 with bad export cookie 8868632568389949126 [ 2554.756579] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2554.768821] LustreError: 103880:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2554.973955] Lustre: server umount lustre-MDT0001 complete [ 2568.267554] Lustre: server umount lustre-OST0000 complete [ 2574.765605] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2575.164720] LustreError: 106137:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2575.236402] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2579.592350] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 2580.450281] LustreError: 106138:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2585.571525] LustreError: 106137:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2587.531210] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2587.898090] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2592.010730] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 2598.626478] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2603.768455] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2609.632341] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2609.995859] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2616.120281] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 2620.210793] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3 to 0x240000400:65) [ 2622.665956] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2628.576476] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2628.584633] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2628.597207] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2628.602702] Lustre: Skipped 1 previous similar message [ 2630.624336] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2630.629327] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2630.630322] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2630.643296] Lustre: Skipped 1 previous similar message [ 2634.050657] Lustre: server umount lustre-OST0000 complete [ 2639.328603] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2639.336149] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2639.345426] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2643.551031] Lustre: server umount lustre-MDT0000 complete [ 2645.988160] LustreError: 106138:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2646.007336] LustreError: 106138:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2646.902370] LustreError: 106117:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789998464 with bad export cookie 8868632568389949819 [ 2646.910148] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2646.918431] LustreError: 106117:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2647.251698] Lustre: server umount lustre-MDT0001 complete [ 2657.076653] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 2659.851936] Key type lgssc unregistered [ 2660.135040] LNet: 109669:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2660.143083] LNetError: 109669:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2660.159359] LNet: Removed LNI 192.168.204.128@tcp [ 2660.878697] Key type .llcrypt unregistered [ 2660.881957] Key type ._llcrypt unregistered [ 2680.548539] Key type ._llcrypt registered [ 2680.551512] Key type .llcrypt registered [ 2680.660232] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 2681.614258] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2681.643441] alg: No test for adler32 (adler32-zlib) [ 2682.694977] Lustre: Lustre: Build Version: 2.17.58_63_g3df9749 [ 2682.976751] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 2684.671607] Key type lgssc registered [ 2685.813256] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2693.327339] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2693.338801] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2694.914417] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2699.165251] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 2704.353727] LustreError: 111080:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2704.373083] LustreError: 111080:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2707.347781] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2707.687052] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2711.795945] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 2718.970680] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2724.052734] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2730.073143] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2730.480499] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2736.129501] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 2739.701175] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:67 to 0x240000400:97) [ 2742.961031] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2747.747340] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 2750.706569] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 2755.053146] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2755.059895] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2755.077698] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2755.087719] Lustre: Skipped 1 previous similar message [ 2756.475448] Lustre: server umount lustre-OST0000 complete [ 2764.256345] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2764.279289] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2764.297414] Lustre: Skipped 1 previous similar message [ 2764.308846] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2765.288096] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2766.966745] Lustre: server umount lustre-MDT0000 complete [ 2770.402532] LustreError: 113345:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2770.417867] LustreError: 113345:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2770.953718] LustreError: 112922:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789998588 with bad export cookie 13291660625826300878 [ 2770.955582] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2770.965533] LustreError: 112922:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2771.385625] Lustre: server umount lustre-MDT0001 complete [ 2783.195389] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 2785.943963] Key type lgssc unregistered [ 2786.231847] LNet: 114716:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2786.237232] LNetError: 114716:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2786.251056] LNet: Removed LNI 192.168.204.128@tcp [ 2787.205654] Key type .llcrypt unregistered [ 2787.208825] Key type ._llcrypt unregistered [ 2803.829314] Key type ._llcrypt registered [ 2803.833663] Key type .llcrypt registered [ 2803.954987] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 2805.050941] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2805.075576] alg: No test for adler32 (adler32-zlib) [ 2806.122948] Lustre: Lustre: Build Version: 2.17.58_63_g3df9749 [ 2806.332122] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 2808.025750] Key type lgssc registered [ 2809.196691] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2817.182838] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2817.213430] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2818.747842] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2823.387993] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 2828.268200] LustreError: 115954:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2828.284535] LustreError: 115954:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2832.106198] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2832.466344] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2836.683719] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 2844.364427] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2850.276617] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2857.036115] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2857.615146] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2863.745513] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:99 to 0x240000400:129) [ 2864.017109] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 2872.081822] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2884.584306] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2884.588222] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2884.590678] LustreError: Skipped 1 previous similar message [ 2884.601453] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2884.608695] Lustre: Skipped 1 previous similar message [ 2886.641324] Lustre: server umount lustre-OST0000 complete [ 2893.793935] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2893.802604] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2893.824741] Lustre: Skipped 1 previous similar message [ 2893.832303] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2894.821197] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2894.839928] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2894.846642] Lustre: Skipped 1 previous similar message [ 2898.501229] Lustre: server umount lustre-MDT0000 complete [ 2899.936332] LustreError: 116988:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2899.953198] LustreError: 116988:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2901.553946] LustreError: 115935:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789998719 with bad export cookie 12922343747847545564 [ 2901.568335] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2901.859498] Lustre: server umount lustre-MDT0001 complete [ 2911.725798] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 2913.921190] Key type lgssc unregistered [ 2914.168437] LNet: 119485:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2914.181414] LNetError: 119485:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2914.202393] LNet: Removed LNI 192.168.204.128@tcp [ 2914.886326] Key type .llcrypt unregistered [ 2914.889310] Key type ._llcrypt unregistered [ 2935.110969] Key type ._llcrypt registered [ 2935.113584] Key type .llcrypt registered [ 2935.247754] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 2936.194738] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2936.226455] alg: No test for adler32 (adler32-zlib) [ 2937.263485] Lustre: Lustre: Build Version: 2.17.58_63_g3df9749 [ 2937.459993] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 2939.128379] Key type lgssc registered [ 2940.277099] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2946.801025] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2946.817872] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2948.235122] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2952.234048] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 2958.818737] LustreError: 120908:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2958.835152] LustreError: 120908:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2959.547800] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2959.874349] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2963.919720] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 2970.340378] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2975.116692] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2980.635489] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2980.961625] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2986.141720] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 2992.416232] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2993.285813] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:131 to 0x240000400:161) [ 3001.761776] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3004.658422] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3004.662531] Lustre: Skipped 2 previous similar messages [ 3008.048217] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3008.055116] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3008.058952] Lustre: Skipped 2 previous similar messages [ 3008.092441] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3014.505371] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.204.28@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3014.512719] Lustre: Skipped 2 previous similar messages [ 3014.923549] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3014.939426] Lustre: Skipped 1 previous similar message [ 3014.955833] Lustre: lustre-MDT0000: nosquash_nids set to 2@gni 0@lo 192.168.204.28@tcp 192.168.0.[2,10]@tcp [ 3014.964148] Lustre: Skipped 1 previous similar message [ 3021.858059] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.204.28/24@tcp in log lustre-MDT0000 [ 3021.865593] Lustre: Skipped 2 previous similar messages [ 3028.004069] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.28/24@tcp [ 3034.597902] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3034.604633] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3034.609602] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3034.613749] Lustre: Skipped 1 previous similar message [ 3038.984659] Lustre: server umount lustre-OST0000 complete [ 3044.839233] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3044.856487] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3044.863392] Lustre: Skipped 1 previous similar message [ 3044.871717] Lustre: Skipped 1 previous similar message [ 3048.133840] Lustre: server umount lustre-MDT0000 complete [ 3049.953805] LustreError: 122746:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3049.972667] LustreError: 122746:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3051.081683] LustreError: 123536:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789998869 with bad export cookie 8833483304217861428 [ 3051.084425] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3051.097390] LustreError: 123536:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3051.365228] Lustre: server umount lustre-MDT0001 complete [ 3060.088660] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 3062.645299] Key type lgssc unregistered [ 3062.968804] LNet: 125647:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3062.987870] LNetError: 125647:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3064.037702] LNet: Removed LNI 192.168.204.128@tcp [ 3064.804186] Key type .llcrypt unregistered [ 3064.806640] Key type ._llcrypt unregistered [ 3086.070569] Key type ._llcrypt registered [ 3086.072546] Key type .llcrypt registered [ 3086.160908] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 3086.923748] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3086.982818] alg: No test for adler32 (adler32-zlib) [ 3087.952349] Lustre: Lustre: Build Version: 2.17.58_63_g3df9749 [ 3088.164589] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 3089.839298] Key type lgssc registered [ 3090.856466] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3096.887461] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3096.900293] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3098.222400] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3098.235309] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.204.28/24@tcp [ 3098.258829] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3101.292847] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 3107.521166] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3107.748474] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3107.768453] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.28/24@tcp [ 3107.833082] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3111.067954] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 3116.699447] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3121.944687] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3127.158864] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3127.462733] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3132.475471] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 3138.142556] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3143.852058] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:169 to 0x240000400:193) [ 3154.401191] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3154.405252] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3154.407739] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3154.412655] LustreError: Skipped 1 previous similar message [ 3154.425971] Lustre: Skipped 1 previous similar message [ 3159.532159] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3159.538171] Lustre: Skipped 2 previous similar messages [ 3160.335828] Lustre: server umount lustre-OST0000 complete [ 3164.130907] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3164.137474] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3164.149813] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3169.537367] Lustre: server umount lustre-MDT0000 complete [ 3169.765715] LustreError: 127205:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3169.780919] LustreError: 127205:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 3172.685522] LustreError: 127187:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789998990 with bad export cookie 7336370843163055218 [ 3172.688918] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3172.695649] LustreError: 127187:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3172.995688] Lustre: server umount lustre-MDT0001 complete [ 3182.051205] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 3184.443898] Key type lgssc unregistered [ 3184.709779] LNet: 130810:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3184.714481] LNetError: 130810:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3184.733463] LNet: Removed LNI 192.168.204.128@tcp [ 3185.358837] Key type .llcrypt unregistered [ 3185.361203] Key type ._llcrypt unregistered [ 3204.816648] Key type ._llcrypt registered [ 3204.819686] Key type .llcrypt registered [ 3204.888281] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_hostid [ 3215.052862] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 3215.526504] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3215.537066] alg: No test for adler32 (adler32-zlib) [ 3216.518937] Lustre: Lustre: Build Version: 2.17.58_63_g3df9749 [ 3216.669543] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 3218.327685] Key type lgssc registered [ 3219.272641] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3276.333269] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3276.360939] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3277.718784] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3277.810972] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3277.936695] Lustre: lustre-MDT0000: new disk, initializing [ 3278.119404] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3278.158475] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3282.836101] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 3293.304502] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3293.405605] Lustre: 134204:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3293.458272] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3293.466208] Lustre: Skipped 1 previous similar message [ 3293.523634] Lustre: lustre-MDT0001: new disk, initializing [ 3293.683526] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3293.723065] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3293.731675] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3298.755872] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 3308.668621] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3315.439577] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3322.522421] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3322.784130] Lustre: lustre-OST0000: new disk, initializing [ 3322.787549] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3322.794910] Lustre: 136177:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3322.858985] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3328.373869] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 3331.653818] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3331.666223] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3331.765459] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3337.442664] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3341.382992] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3341.611434] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3346.713445] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3346.993637] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3352.548020] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3352.558876] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3352.563392] LustreError: Skipped 1 previous similar message [ 3352.563665] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3352.576647] Lustre: Skipped 1 previous similar message [ 3357.667298] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3357.678797] Lustre: Skipped 2 previous similar messages [ 3362.797667] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3362.813782] Lustre: Skipped 1 previous similar message [ 3363.295913] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3363.504818] Lustre: server umount lustre-OST0000 complete [ 3367.907528] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3367.920103] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3367.926442] Lustre: Skipped 1 previous similar message [ 3367.946948] Lustre: Skipped 1 previous similar message [ 3373.031143] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3373.047085] Lustre: Skipped 1 previous similar message [ 3373.915861] Lustre: server umount lustre-MDT0000 complete [ 3377.788536] LustreError: 134199:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789999195 with bad export cookie 9596831114760683778 [ 3377.796380] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3377.810262] LustreError: 134199:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3378.306978] Lustre: server umount lustre-MDT0001 complete [ 3387.982743] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3388.699833] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3394.019625] LustreError: 138395:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3394.052622] LustreError: 138395:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3394.109458] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 3398.115396] LustreError: 138394:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3402.846704] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3403.375647] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3408.264935] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 3415.957373] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3422.236886] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3429.849657] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3430.336032] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3437.295331] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 3445.921619] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3451.315229] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3451.629770] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3456.906781] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3457.169163] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3467.069605] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3467.237458] Lustre: lustre-OST0001: new disk, initializing [ 3467.241125] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3467.246499] Lustre: 141782:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3467.358526] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3474.470970] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 3477.132851] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3477.140273] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3477.256600] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3484.361408] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3484.601844] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3490.679434] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3490.886317] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3497.671563] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8aa45171e800.ost_server_uuid 50 [ 3499.758310] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8aa45171e800.ost_server_uuid in IDLE state after 0 sec [ 3513.312281] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3513.318632] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3513.327034] LustreError: Skipped 1 previous similar message [ 3513.338387] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3513.341934] Lustre: Skipped 1 previous similar message [ 3513.364145] Lustre: Skipped 1 previous similar message [ 3515.382697] Lustre: server umount lustre-OST0001 complete [ 3518.469679] LustreError: 142037:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3518.497167] LustreError: 142037:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3520.484085] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3520.491938] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3520.499097] LustreError: Skipped 1 previous similar message [ 3520.530608] Lustre: Skipped 1 previous similar message [ 3523.559637] LustreError: 142037:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3523.585428] LustreError: 142037:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3528.690137] LustreError: 140216:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3528.724128] LustreError: 140216:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3533.793420] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3533.810551] Lustre: Skipped 7 previous similar messages [ 3534.303418] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3534.676272] Lustre: server umount lustre-OST0000 complete [ 3538.912490] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3538.923861] Lustre: Skipped 1 previous similar message [ 3544.582689] Lustre: server umount lustre-MDT0000 complete [ 3549.155278] LustreError: 139761:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3549.192308] LustreError: 139761:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 3549.553609] LustreError: 138377:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789999367 with bad export cookie 9596831114760684660 [ 3549.558907] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3549.570747] LustreError: 138377:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3550.193674] Lustre: server umount lustre-MDT0001 complete [ 3573.067206] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 3576.661759] Key type lgssc unregistered [ 3577.028163] LNet: 144785:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3577.039492] LNetError: 144785:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3578.085773] LNet: Removed LNI 192.168.204.128@tcp [ 3579.087084] Key type .llcrypt unregistered [ 3579.089251] Key type ._llcrypt unregistered [ 3623.310996] Key type ._llcrypt registered [ 3623.314249] Key type .llcrypt registered [ 3623.542509] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_hostid [ 3641.433629] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 3642.530104] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3642.611482] alg: No test for adler32 (adler32-zlib) [ 3643.861812] Lustre: Lustre: Build Version: 2.17.58_63_g3df9749 [ 3644.099906] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 3645.831569] Key type lgssc registered [ 3647.294399] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3685.003186] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3685.058056] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3686.341743] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3686.367580] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3686.438898] Lustre: lustre-MDT0000: new disk, initializing [ 3686.529394] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3686.581949] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3691.653646] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 3704.077356] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3704.240071] Lustre: 148958:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3704.313726] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3704.319579] Lustre: Skipped 1 previous similar message [ 3704.374921] Lustre: lustre-MDT0001: new disk, initializing [ 3704.469311] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3704.523408] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3704.541908] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3709.409593] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 3719.612580] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3725.352302] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3732.599849] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3732.881441] Lustre: lustre-OST0000: new disk, initializing [ 3732.887617] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3732.901616] Lustre: 150926:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3733.034062] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3740.270428] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 3740.721122] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3740.731725] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3740.848747] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3750.136834] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3755.520922] Lustre: Failing over lustre-OST0000 [ 3755.792356] Lustre: server umount lustre-OST0000 complete [ 3772.895117] Lustre: 147174:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789999574/real 1789999574] req@0000000004918a1c x1876950475901440/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1789999590 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3772.895769] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3772.922969] Lustre: 147174:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 3772.936137] Lustre: Skipped 1 previous similar message [ 3773.919625] Lustre: 147173:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789999575/real 1789999575] req@0000000013c2aba9 x1876950475901568/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1789999591 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3774.833482] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3775.356329] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3775.376755] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 3778.399873] Lustre: 147173:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789999579/real 1789999579] req@000000001dd1eef9 x1876950475902592/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789999595 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3781.296138] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 3782.563497] Lustre: 147173:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789999584/real 1789999584] req@00000000798dc590 x1876950475903616/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789999600 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3782.580653] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 3782.589244] Lustre: 147173:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 3784.730203] Lustre: Failing over lustre-MDT0000 [ 3784.766603] LustreError: 147170:0:(import.c:752:ptlrpc_connect_import_locked()) can't connect to a closed import [ 3784.849444] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.28@tcp (stopping) [ 3784.867851] Lustre: lustre-OST0000: Recovery over after 0:02, of 3 clients 3 recovered and 0 were evicted. [ 3784.869702] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3785.126534] Lustre: server umount lustre-MDT0000 complete [ 3786.724402] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3786.734986] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3787.760048] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3787.780338] Lustre: Skipped 1 previous similar message [ 3788.767214] Lustre: 147173:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789999589/real 1789999589] req@000000006e6cd667 x1876950475904512/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789999605 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3788.798320] Lustre: 147173:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 3792.868755] LustreError: 148964:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3792.894044] LustreError: 148964:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 8 previous similar messages [ 3795.057200] LustreError: 148966:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3797.991332] LustreError: 148969:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3798.012599] LustreError: 148969:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3800.178924] LustreError: 148966:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3803.105452] Lustre: 147172:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789999605/real 1789999605] req@00000000a1c332a1 x1876950475913984/t0(0) o400->MGC192.168.204.128@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789999621 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3803.164757] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3804.466853] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3805.303129] LustreError: 150034:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3805.332487] LustreError: 150034:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 3813.347910] LustreError: 148969:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3813.381105] LustreError: 148969:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 3813.690189] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 3813.780177] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3813.849809] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3815.551219] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3818.176121] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 3818.983241] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3819.006181] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 3819.056677] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 3825.187627] Lustre: server umount lustre-OST0000 complete [ 3829.220272] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3829.228697] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3829.233090] Lustre: Skipped 1 previous similar message [ 3829.244482] Lustre: Skipped 1 previous similar message [ 3834.345734] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3834.367346] Lustre: Skipped 1 previous similar message [ 3838.501420] Lustre: server umount lustre-MDT0000 complete [ 3839.457911] LustreError: 148970:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3839.475339] LustreError: 148970:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3842.116676] LustreError: 149753:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789999660 with bad export cookie 10394506229850052130 [ 3842.124594] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3842.127411] LustreError: 149753:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3853.034206] Lustre: server umount lustre-MDT0001 complete [ 3864.631428] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 3867.604783] Key type lgssc unregistered [ 3867.951113] LNet: 154584:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3867.967843] LNetError: 154584:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3867.994367] LNet: Removed LNI 192.168.204.128@tcp [ 3868.956447] Key type .llcrypt unregistered [ 3868.958685] Key type ._llcrypt unregistered [ 3906.819702] Key type ._llcrypt registered [ 3906.823650] Key type .llcrypt registered [ 3906.948731] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 3908.298282] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3908.394040] alg: No test for adler32 (adler32-zlib) [ 3909.734224] Lustre: Lustre: Build Version: 2.17.58_63_g3df9749 [ 3910.159254] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 3911.903264] Key type lgssc registered [ 3913.465959] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3922.631145] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3922.667107] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3924.213240] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3929.097447] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 3934.690328] LustreError: 156120:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3934.701284] LustreError: 156120:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3938.125171] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3939.811363] LustreError: 156119:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3941.822343] LustreError: 156148:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 3941.838240] LustreError: 156148:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 3941.848680] LustreError: 156148:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 17, retries 0, failed: rc = -5 [ 3942.336149] Lustre: server umount lustre-MDT0000 complete [ 3957.058890] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_hostid [ 3966.976695] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 4008.532484] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4008.844672] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4008.892375] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4008.941358] Lustre: lustre-MDT0000: new disk, initializing [ 4009.001056] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4009.015273] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4013.770756] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 4023.067454] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4023.178935] Lustre: 159453:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4023.223691] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4023.226941] Lustre: Skipped 1 previous similar message [ 4023.281218] Lustre: lustre-MDT0001: new disk, initializing [ 4023.375747] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4023.423902] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4023.448242] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4027.837653] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 4038.275367] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4044.732878] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4052.485675] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4053.015533] Lustre: lustre-OST0000: new disk, initializing [ 4053.029875] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4053.045626] Lustre: 161419:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4053.222107] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4059.723423] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 4059.778894] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4059.795759] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4059.935148] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4069.034386] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4073.576847] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4075.123925] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 10:11:30 (1789999890) [ 4309.987708] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4310.001347] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4310.020835] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4311.007802] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4311.008781] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4311.014362] Lustre: Skipped 1 previous similar message [ 4311.034286] Lustre: Skipped 1 previous similar message [ 4313.973589] Lustre: server umount lustre-MDT0000 complete [ 4316.128666] LustreError: 163190:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4316.167712] LustreError: 163190:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4318.329846] LustreError: 159446:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790000136 with bad export cookie 532647919601892125 [ 4318.329846] LustreError: 159445:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790000136 with bad export cookie 532647919601892125 [ 4318.330810] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4318.337946] LustreError: 159446:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4321.248463] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4321.249145] LustreError: 159460:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4321.266945] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4321.266955] Lustre: Skipped 1 previous similar message [ 4323.307417] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4324.848280] Lustre: server umount lustre-MDT0001 complete [ 4335.372072] Lustre: server umount lustre-OST0000 complete [ 4346.117554] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_hostid [ 4356.744864] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 4393.786437] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4394.050321] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4394.092728] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4394.190407] Lustre: lustre-MDT0000: new disk, initializing [ 4394.312674] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4394.347847] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4399.548551] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 4410.877124] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4411.061442] Lustre: 166083:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4411.136567] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4411.141455] Lustre: Skipped 1 previous similar message [ 4411.231278] Lustre: lustre-MDT0001: new disk, initializing [ 4411.346863] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4411.400447] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4411.418122] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4416.947549] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 4427.563500] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4434.092246] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4441.191552] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4441.579543] Lustre: lustre-OST0000: new disk, initializing [ 4441.583885] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4441.601457] Lustre: 168055:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4441.726641] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4447.867748] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4447.880349] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4448.071908] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4449.063469] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 4461.192649] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4467.268213] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4467.577726] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4472.510811] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4472.806150] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4478.950787] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4478.958435] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4478.969394] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4478.974246] Lustre: Skipped 1 previous similar message [ 4489.184317] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4489.185961] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4489.207313] Lustre: Skipped 3 previous similar messages [ 4489.331236] Lustre: server umount lustre-OST0000 complete [ 4493.283846] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4493.295309] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4493.318550] Lustre: Skipped 1 previous similar message [ 4499.615243] Lustre: server umount lustre-MDT0000 complete [ 4503.255420] LustreError: 166076:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790000321 with bad export cookie 532647919602050948 [ 4503.259569] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4503.273979] LustreError: 166076:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4503.704194] Lustre: server umount lustre-MDT0001 complete [ 4512.400712] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 10:18:47 (1790000327) [ 4529.970891] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_hostid [ 4539.632906] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 4576.350529] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4576.596050] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4576.645867] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4576.733706] Lustre: lustre-MDT0000: new disk, initializing [ 4576.819070] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4576.833720] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4580.460606] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 4590.829505] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4591.038664] Lustre: 172817:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4591.186648] LustreError: 172827:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4591.219963] LustreError: 172827:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4591.554217] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4597.000559] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 4606.706486] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4611.581207] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4617.386345] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4617.694531] Lustre: lustre-OST0000: new disk, initializing [ 4617.698797] Lustre: Skipped 1 previous similar message [ 4617.721748] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4617.731841] Lustre: Skipped 2 previous similar messages [ 4617.756552] Lustre: 174796:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4625.307164] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 4625.557080] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4625.566693] Lustre: Skipped 1 previous similar message [ 4625.574465] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4625.807507] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4636.930585] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4646.370987] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4646.371902] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4646.372223] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4646.372227] Lustre: Skipped 5 previous similar messages [ 4646.387596] Lustre: Skipped 2 previous similar messages [ 4646.404170] LustreError: Skipped 1 previous similar message [ 4658.143761] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4658.361201] Lustre: server umount lustre-OST0000 complete [ 4667.963333] Lustre: server umount lustre-MDT0000 complete [ 4671.519685] LustreError: 172808:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790000489 with bad export cookie 532647919602051837 [ 4671.520750] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4671.531610] LustreError: 172808:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4671.951385] Lustre: server umount lustre-MDT0001 complete [ 4680.819057] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 10:21:36 (1790000496) [ 4698.531577] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_hostid [ 4707.270811] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 4737.927443] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4738.121588] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4738.141171] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4738.214457] Lustre: lustre-MDT0000: new disk, initializing [ 4738.302933] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4738.310089] Lustre: Skipped 2 previous similar messages [ 4738.333292] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4742.046488] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 4752.460632] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4752.637850] Lustre: 179121:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4752.902788] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4758.463298] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 4767.724847] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4773.640366] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4780.019396] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4780.273819] Lustre: 181097:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4785.886238] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 4787.285197] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4787.426439] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4793.727692] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4803.052670] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4803.052044] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4803.066573] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4803.081438] Lustre: Skipped 3 previous similar messages [ 4803.104683] Lustre: Skipped 10 previous similar messages [ 4805.695384] Lustre: server umount lustre-OST0000 complete [ 4815.509822] Lustre: server umount lustre-MDT0000 complete [ 4818.400517] LustreError: 179132:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4818.411792] LustreError: 179132:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4818.695584] LustreError: 179112:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790000636 with bad export cookie 532647919602052922 [ 4818.699262] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4818.710337] LustreError: 179112:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4819.023024] Lustre: server umount lustre-MDT0001 complete [ 4827.627879] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 4830.244746] Key type lgssc unregistered [ 4830.518990] LNet: 182951:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4830.541168] LNetError: 182951:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4831.589530] LNet: Removed LNI 192.168.204.128@tcp [ 4832.472546] Key type .llcrypt unregistered [ 4832.474834] Key type ._llcrypt unregistered [ 4857.688283] Key type ._llcrypt registered [ 4857.690268] Key type .llcrypt registered [ 4857.800668] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 4859.318723] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4859.378390] alg: No test for adler32 (adler32-zlib) [ 4860.638846] Lustre: Lustre: Build Version: 2.17.58_63_g3df9749 [ 4860.889895] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 4862.583254] Key type lgssc registered [ 4864.053893] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4872.000625] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4872.015757] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4873.595658] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4879.185235] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 4883.938826] LustreError: 184384:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4883.946446] LustreError: 184384:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4887.537475] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4887.902470] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4892.452441] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 4901.763586] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4907.969932] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4914.934299] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4915.335350] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4918.454435] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4921.390647] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 4928.315801] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4949.475424] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4949.484252] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4949.497385] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4953.570102] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4953.575596] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4953.576258] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4955.822683] Lustre: server umount lustre-OST0000 complete [ 4963.812324] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4963.817747] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4963.833153] Lustre: Skipped 1 previous similar message [ 4963.838716] Lustre: Skipped 1 previous similar message [ 4966.311023] Lustre: server umount lustre-MDT0000 complete [ 4968.930959] LustreError: 185538:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4968.957646] LustreError: 185538:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4970.415903] LustreError: 184367:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790000788 with bad export cookie 10099744985374796240 [ 4970.422697] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4970.435658] LustreError: 184367:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4970.850428] Lustre: server umount lustre-MDT0001 complete [ 4983.938434] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 4987.180968] Key type lgssc unregistered [ 4987.593781] LNet: 187939:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4987.609318] LNetError: 187939:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4987.627430] LNet: Removed LNI 192.168.204.128@tcp [ 4988.668895] Key type .llcrypt unregistered [ 4988.671525] Key type ._llcrypt unregistered [ 5015.318696] Key type ._llcrypt registered [ 5015.320755] Key type .llcrypt registered [ 5015.417978] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 5016.743709] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5016.821069] alg: No test for adler32 (adler32-zlib) [ 5017.937101] Lustre: Lustre: Build Version: 2.17.58_63_g3df9749 [ 5018.157662] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 5019.839550] Key type lgssc registered [ 5021.314924] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5029.805345] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5029.870403] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5031.452612] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5036.836221] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 5041.633577] LustreError: 189371:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5041.651111] LustreError: 189371:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5044.609894] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5045.009659] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5049.636062] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 5059.013537] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5066.263243] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5072.912874] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5073.294602] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5076.424408] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5080.002362] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 5089.001170] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5094.884829] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5094.900963] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5094.905801] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5094.916024] Lustre: Skipped 1 previous similar message [ 5096.086811] Lustre: lustre-OST0000: Not available for connect from 192.168.204.28@tcp (stopping) [ 5100.007070] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5100.014465] Lustre: Skipped 1 previous similar message [ 5103.012274] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5103.019378] Lustre: Skipped 2 previous similar messages [ 5108.195146] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5108.206184] Lustre: Skipped 2 previous similar messages [ 5109.215152] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5109.513525] Lustre: server umount lustre-OST0000 complete [ 5117.207243] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff8aa451951000.ost_server_uuid 50 [ 5119.337843] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5133.794144] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5133.806099] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5133.809620] Lustre: Skipped 2 previous similar messages [ 5133.824873] Lustre: Skipped 1 previous similar message [ 5138.804788] Lustre: server umount lustre-MDT0000 complete [ 5138.918333] LustreError: 190099:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5138.946496] LustreError: 190099:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5139.944440] LustreError: 189366:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5139.967660] LustreError: 189366:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5143.051560] LustreError: 189351:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790000961 with bad export cookie 6031721419454523683 [ 5143.061874] LustreError: 189351:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5143.066439] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5143.409671] Lustre: server umount lustre-MDT0001 complete [ 5151.624976] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 10:29:27 (1790000967) [ 5157.596584] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5157.932506] LustreError: 193291:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5157.992370] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5161.940183] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 5163.496315] LustreError: 193292:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5170.670541] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5171.122407] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5175.263416] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 5182.715693] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5188.407697] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5195.022663] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5195.323608] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5201.480516] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 5202.512697] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5211.139260] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5219.887516] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5220.215526] Lustre: lustre-OST0001: new disk, initializing [ 5220.218765] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5220.227150] Lustre: 196130:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5220.317503] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5226.596894] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5226.603975] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5226.733199] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5227.370413] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 5237.355784] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5242.342410] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5242.345139] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5242.360841] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5242.361739] Lustre: Skipped 1 previous similar message [ 5242.376614] Lustre: Skipped 1 previous similar message [ 5245.991469] Lustre: server umount lustre-OST0000 complete [ 5247.458792] LustreError: 195111:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5247.481369] LustreError: 195111:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5253.350341] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5253.550354] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5257.756129] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5257.968033] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5263.481808] LustreError: 195114:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5263.510749] LustreError: 195114:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 9 previous similar messages [ 5273.062252] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5273.068556] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5273.072262] LustreError: Skipped 1 previous similar message [ 5273.098199] Lustre: Skipped 1 previous similar message [ 5277.363670] Lustre: server umount lustre-OST0001 complete [ 5283.298078] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5283.302538] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5283.317324] Lustre: Skipped 1 previous similar message [ 5283.335672] Lustre: Skipped 4 previous similar messages [ 5287.386781] Lustre: server umount lustre-MDT0000 complete [ 5290.879852] LustreError: 193274:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790001108 with bad export cookie 6031721419454524761 [ 5290.888645] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5290.892395] LustreError: 193274:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5291.338171] Lustre: server umount lustre-MDT0001 complete [ 5314.650874] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 10:32:09 (1790001129) [ 5320.508337] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5320.791383] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5320.838587] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5321.060493] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5325.050192] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 5333.793331] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5333.937207] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5333.991596] Lustre: 199570:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5338.542461] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 5345.394506] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5350.627505] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5357.908451] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5358.143050] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5366.159051] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 5366.934366] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 5373.887959] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5379.543210] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5379.622567] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5385.445771] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 5386.068882] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 5393.820597] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5397.472605] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5397.482406] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5397.504690] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5403.160289] Lustre: server umount lustre-OST0000 complete [ 5404.788152] LustreError: 201422:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5404.810778] LustreError: 201422:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 5421.545629] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5421.553475] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5421.566645] LustreError: Skipped 1 previous similar message [ 5421.595548] Lustre: Skipped 1 previous similar message [ 5424.736297] Lustre: server umount lustre-OST0001 complete [ 5431.776954] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5431.785186] LustreError: Skipped 1 previous similar message [ 5434.567703] Lustre: server umount lustre-MDT0000 complete [ 5438.117145] LustreError: 199563:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790001256 with bad export cookie 6031721419454526168 [ 5438.118990] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5438.134405] LustreError: 199563:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5438.546564] Lustre: server umount lustre-MDT0001 complete [ 5464.388336] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 10:34:39 (1790001279) [ 5481.785579] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_hostid [ 5493.066453] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 5533.079269] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5533.417059] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5533.482436] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5533.613283] Lustre: lustre-MDT0000: new disk, initializing [ 5533.704634] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5533.710292] Lustre: Skipped 3 previous similar messages [ 5533.730327] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5538.994826] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 5551.808316] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5551.969526] Lustre: 207668:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5552.053231] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5552.063255] Lustre: Skipped 1 previous similar message [ 5552.163514] Lustre: lustre-MDT0001: new disk, initializing [ 5552.342286] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5552.355127] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5557.593247] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 5567.408902] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5572.974357] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5580.009617] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5580.283944] Lustre: lustre-OST0000: new disk, initializing [ 5580.287983] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5580.313760] Lustre: 209638:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5586.565609] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 5589.064256] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5589.073629] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5589.170752] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5595.916250] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5601.114980] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5601.329254] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5605.810921] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5606.000386] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5609.964238] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5609.964858] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5609.995869] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5609.995878] Lustre: Skipped 8 previous similar messages [ 5610.001229] Lustre: Skipped 4 previous similar messages [ 5614.529566] Lustre: server umount lustre-OST0000 complete [ 5625.321499] LustreError: 207682:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5625.331159] LustreError: 207682:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 13 previous similar messages [ 5625.772915] Lustre: server umount lustre-MDT0000 complete [ 5630.689491] LustreError: 207663:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790001448 with bad export cookie 6031721419454527953 [ 5630.694953] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5630.697183] LustreError: 207663:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5630.962374] Lustre: server umount lustre-MDT0001 complete [ 5640.128292] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5645.940932] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 5655.618333] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5660.681370] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 5668.080592] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5674.231400] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5680.954425] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5681.280154] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5681.283438] Lustre: Skipped 4 previous similar messages [ 5687.881595] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 5695.472054] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5700.431386] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5700.622259] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5705.413246] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5705.637724] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5711.334232] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5711.341966] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5711.375696] Lustre: Skipped 3 previous similar messages [ 5711.385754] LustreError: Skipped 2 previous similar messages [ 5714.307572] Lustre: server umount lustre-OST0000 complete [ 5722.344731] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5728.275839] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5758.347635] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5765.086841] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 5773.114636] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5776.095936] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 1 sec [ 5776.236827] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5776.268283] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5776.269734] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5776.305242] Lustre: Skipped 1 previous similar message [ 5781.490823] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5787.784173] Lustre: server umount lustre-OST0000 complete [ 5798.107104] Lustre: server umount lustre-MDT0000 complete [ 5801.613661] LustreError: 215969:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790001619 with bad export cookie 6031721419454528835 [ 5801.614355] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5801.625494] LustreError: 215969:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5802.037870] Lustre: server umount lustre-MDT0001 complete [ 5810.434893] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 10:40:26 (1790001626) [ 5816.881896] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5817.370566] LustreError: 217827:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5817.394531] LustreError: 217827:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 5821.406216] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 5829.608524] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5834.824446] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 5845.475700] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5852.625271] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5859.988487] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5867.658279] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 5875.644563] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5879.797353] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5879.978696] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5885.273223] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5885.625730] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5893.250362] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5893.486841] Lustre: lustre-OST0001: new disk, initializing [ 5893.491953] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5893.497675] Lustre: 221196:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5900.348970] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5900.361752] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5900.449018] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5900.606693] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 5911.307088] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5916.752343] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5917.093222] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5922.804445] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5923.063501] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5926.377318] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5926.400078] LustreError: Skipped 1 previous similar message [ 5926.409832] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5926.427616] Lustre: Skipped 4 previous similar messages [ 5926.435389] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5926.443586] Lustre: Skipped 15 previous similar messages [ 5931.540218] Lustre: server umount lustre-OST0001 complete [ 5938.240235] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5938.474102] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5943.874399] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5944.104350] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5973.907155] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5974.326861] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5974.339050] Lustre: Skipped 5 previous similar messages [ 5979.634776] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 5979.643340] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5979.644828] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 5980.559857] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 5987.614510] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5989.367669] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5997.313749] Lustre: server umount lustre-OST0001 complete [ 6028.255184] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6028.745298] Lustre: server umount lustre-OST0000 complete [ 6048.223351] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6048.525827] Lustre: server umount lustre-MDT0000 complete [ 6052.987103] LustreError: 221195:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790001870 with bad export cookie 6031721419454529801 [ 6052.994184] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6053.003780] LustreError: 221195:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6053.014568] LustreError: 188674:0:(import.c:762:ptlrpc_connect_import_locked()) already connecting [ 6059.405324] Lustre: server umount lustre-MDT0001 complete [ 6085.090293] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 10:45:00 (1790001900) [ 6092.132055] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6092.491688] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6092.527340] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6097.629948] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 6106.640734] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6106.719036] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6106.742970] Lustre: 226444:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6106.803270] LustreError: 226456:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6106.815844] LustreError: 226456:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 40 previous similar messages [ 6112.227318] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 6119.972764] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6125.388994] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6132.970264] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6133.189186] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6140.490080] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 6147.966305] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6156.909317] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6156.990374] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6163.778358] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 6171.168804] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6172.419367] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6176.311585] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6176.530609] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6180.982436] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6181.272382] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6186.811731] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8aa44c18f800.ost_server_uuid 50 [ 6188.194789] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8aa44c18f800.ost_server_uuid in IDLE state after 0 sec [ 6188.963670] Lustre: Permanently deactivating lustre-OST0001 [ 6188.981662] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6191.733125] Lustre: Permanently reactivating lustre-OST0001 [ 6195.169648] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6195.175216] LustreError: Skipped 4 previous similar messages [ 6195.182461] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6195.201360] Lustre: Skipped 7 previous similar messages [ 6200.872753] Lustre: server umount lustre-OST0001 complete [ 6210.889731] Lustre: server umount lustre-OST0000 complete [ 6220.841160] Lustre: server umount lustre-MDT0000 complete [ 6226.015156] LustreError: 226438:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790002043 with bad export cookie 6031721419454531089 [ 6226.027830] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6226.033881] LustreError: 226438:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6226.909375] Lustre: server umount lustre-MDT0001 complete [ 6251.607657] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 10:47:46 (1790002066) [ 6260.083323] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6260.459894] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6260.494980] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6260.502500] Lustre: Skipped 5 previous similar messages [ 6265.502810] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 6273.827351] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6273.973882] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6274.020314] Lustre: 233134:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6279.268880] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 6286.473700] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6291.994306] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6299.210547] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6299.453528] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6299.476551] Lustre: Permanently deactivating lustre-OST0000 [ 6299.492612] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 6306.352876] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 6307.894993] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6314.918153] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6322.173513] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6322.287664] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6327.942373] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 6329.102577] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 6337.360235] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6339.163215] Lustre: Permanently reactivating lustre-OST0000 [ 6339.180192] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 6339.192100] Lustre: Skipped 2 previous similar messages [ 6346.405906] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6346.407496] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 39 seconds. I think it's dead, and I am evicting it. exp 00000000cf0c309d, cur 1790002164 deadline 1790002155 last 1790002125 [ 6346.414164] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6346.452802] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6346.464237] Lustre: Skipped 1 previous similar message [ 6346.518588] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6367.199197] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6367.313500] Lustre: server umount lustre-OST0001 complete [ 6377.378580] Lustre: server umount lustre-OST0000 complete [ 6387.653369] Lustre: server umount lustre-MDT0000 complete [ 6392.289592] LustreError: 233126:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790002210 with bad export cookie 6031721419454532734 [ 6392.299194] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6392.308262] LustreError: 233126:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6392.735972] Lustre: server umount lustre-MDT0001 complete [ 6402.636305] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 6405.892512] Key type lgssc unregistered [ 6406.216823] LNet: 238014:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6406.223537] LNetError: 238014:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6406.245681] LNet: Removed LNI 192.168.204.128@tcp [ 6406.992395] Key type .llcrypt unregistered [ 6406.995560] Key type ._llcrypt unregistered [ 6430.321187] Key type ._llcrypt registered [ 6430.323532] Key type .llcrypt registered [ 6430.432621] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 6431.539940] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6431.593343] alg: No test for adler32 (adler32-zlib) [ 6432.806286] Lustre: Lustre: Build Version: 2.17.58_63_g3df9749 [ 6433.226748] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 6435.007217] Key type lgssc registered [ 6436.497530] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6448.069961] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6448.086545] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6449.657643] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6454.447716] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 6459.877334] LustreError: 239507:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6459.910369] LustreError: 239507:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6464.359915] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6464.551443] Lustre: Found index 1 for lustre-MDT0001, updating log [ 6464.623963] Lustre: 239494:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6464.648318] Lustre: Permanently deactivating lustre-MDT0001 [ 6464.685932] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 6465.161562] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6465.706612] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 6465.733319] LustreError: 239537:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 6465.749447] LustreError: 239537:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 6465.756949] LustreError: 239537:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 16, retries 1, failed: rc = -108 [ 6470.808627] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 6479.285361] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6485.354193] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6492.941935] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6493.424657] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6494.445408] LustreError: 241420:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6494.458738] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 6499.818738] LustreError: 241420:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6499.966655] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 6504.936862] LustreError: 241420:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6507.716085] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6510.063448] LustreError: 241347:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6510.100631] LustreError: 241347:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6515.063429] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6515.280251] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6520.315938] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 6520.333276] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 6521.268806] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 6527.642770] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6531.436237] Lustre: Permanently reactivating lustre-MDT0001 [ 6531.448703] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 6531.453560] Lustre: Skipped 2 previous similar messages [ 6534.168563] Lustre: lustre-MDT0001: Client 00f8d494-92f0-4041-bffb-64bd6cd2e65a (at 192.168.204.28@tcp) reconnecting [ 6539.826174] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6542.464937] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6548.745869] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 6549.107089] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 6554.282083] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 6554.566304] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 6557.587770] Lustre: Permanently deactivating lustre-MDT0001 [ 6557.609160] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 6557.617993] Lustre: Skipped 2 previous similar messages [ 6566.065343] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 6575.588078] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6575.591478] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6575.600298] Lustre: Skipped 1 previous similar message [ 6575.616495] Lustre: Skipped 3 previous similar messages [ 6580.713246] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6580.721661] Lustre: Skipped 1 previous similar message [ 6581.572352] Lustre: server umount lustre-MDT0000 complete [ 6585.049747] LustreError: 239486:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790002403 with bad export cookie 7558090729380716360 [ 6585.058829] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6585.061818] LustreError: 239486:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6585.355309] Lustre: server umount lustre-MDT0001 complete [ 6594.638272] Lustre: server umount lustre-OST0000 complete [ 6608.779512] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 10:53:44 (1790002424) [ 6620.282033] Lustre: server umount lustre-OST0001 complete [ 6627.178430] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_hostid [ 6635.607598] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 6677.781339] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6678.107304] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6678.114849] Lustre: Skipped 2 previous similar messages [ 6678.129531] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6678.215981] Lustre: lustre-MDT0000: new disk, initializing [ 6678.311659] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6678.332748] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6684.413361] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 6696.852033] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6696.967304] Lustre: 247815:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6697.050613] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6697.062501] Lustre: Skipped 1 previous similar message [ 6697.231811] Lustre: lustre-MDT0001: new disk, initializing [ 6697.362525] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6697.403324] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6697.429795] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6701.886463] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 6712.940967] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6719.442497] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6726.228480] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6726.604175] Lustre: lustre-OST0000: new disk, initializing [ 6726.616060] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6726.631663] Lustre: 249784:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6726.755709] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6733.038368] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 6735.990047] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6736.000561] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6736.157537] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6742.576901] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6744.533498] LustreError: 249438:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 6753.067490] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6753.246924] Lustre: lustre-OST0001: new disk, initializing [ 6753.251419] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6753.258412] Lustre: 250895:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6754.556490] LustreError: 249438:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 6754.607272] LustreError: 247822:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 6760.589040] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6760.604915] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6760.613068] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 6760.711441] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6764.624879] LustreError: 247822:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 6771.343088] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6772.980519] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6776.305252] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6776.314909] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6776.325980] Lustre: Skipped 2 previous similar messages [ 6776.334319] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6780.837942] Lustre: server umount lustre-OST0001 complete [ 6781.414861] LustreError: 250258:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6781.427976] LustreError: 250258:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 6801.892671] LustreError: 250258:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6801.893431] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6801.899465] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6801.899474] Lustre: Skipped 2 previous similar messages [ 6801.915554] LustreError: 250258:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 8 previous similar messages [ 6801.930462] Lustre: Skipped 2 previous similar messages [ 6804.564829] Lustre: server umount lustre-OST0000 complete [ 6810.080562] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6810.087811] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6810.113934] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6810.119305] Lustre: Skipped 1 previous similar message [ 6813.955630] Lustre: server umount lustre-MDT0000 complete [ 6818.027328] LustreError: 247805:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790002635 with bad export cookie 7558090729380718957 [ 6818.031182] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6818.038093] LustreError: 247805:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6824.632559] Lustre: server umount lustre-MDT0001 complete [ 6837.479770] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 6840.452780] Key type lgssc unregistered [ 6840.720876] LNet: 252930:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6840.738747] LNetError: 252930:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6840.763163] LNet: Removed LNI 192.168.204.128@tcp [ 6841.651884] Key type .llcrypt unregistered [ 6841.655531] Key type ._llcrypt unregistered [ 6882.845648] Key type ._llcrypt registered [ 6882.849380] Key type .llcrypt registered [ 6882.966795] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 6883.988354] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6884.005689] alg: No test for adler32 (adler32-zlib) [ 6885.035217] Lustre: Lustre: Build Version: 2.17.58_63_g3df9749 [ 6885.256435] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 6886.943352] Key type lgssc registered [ 6888.421631] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6897.294679] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6897.356136] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6898.828460] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6898.851378] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6899.129412] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6905.842816] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 6915.280441] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6915.416085] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6915.458393] Lustre: 255150:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6915.731722] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6920.461346] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 6928.101588] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6935.057245] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6941.801699] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6942.000962] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6942.172870] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6947.975524] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 6948.783153] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 6956.059821] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7004.128287] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7004.137668] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7004.146282] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7004.641438] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7004.647054] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7004.668534] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7009.440826] Lustre: server umount lustre-OST0000 complete [ 7014.649109] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7028.199763] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7028.605361] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7035.582652] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 7042.099809] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7044.973918] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7045.008546] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7045.016614] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7045.045270] Lustre: Skipped 1 previous similar message [ 7080.933239] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7080.933594] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7080.938501] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7080.951539] Lustre: Skipped 1 previous similar message [ 7080.966684] LustreError: Skipped 1 previous similar message [ 7086.048477] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7086.056016] Lustre: Skipped 2 previous similar messages [ 7087.003584] Lustre: server umount lustre-OST0000 complete [ 7091.169643] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7091.170942] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7091.197058] Lustre: Skipped 1 previous similar message [ 7091.205647] Lustre: Skipped 1 previous similar message [ 7096.759974] Lustre: server umount lustre-MDT0000 complete [ 7100.109920] LustreError: 255144:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790002918 with bad export cookie 4578058318260370407 [ 7100.120820] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7100.133864] LustreError: 255144:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7100.472858] Lustre: server umount lustre-MDT0001 complete [ 7111.463937] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 7114.932815] Key type lgssc unregistered [ 7115.342232] LNet: 260584:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7115.346950] LNetError: 260584:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7115.370587] LNet: Removed LNI 192.168.204.128@tcp [ 7116.260182] Key type .llcrypt unregistered [ 7116.262321] Key type ._llcrypt unregistered [ 7140.299211] Key type ._llcrypt registered [ 7140.306262] Key type .llcrypt registered [ 7140.457725] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 7141.681682] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7141.706455] alg: No test for adler32 (adler32-zlib) [ 7142.731419] Lustre: Lustre: Build Version: 2.17.58_63_g3df9749 [ 7142.955928] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 7144.719203] Key type lgssc registered [ 7146.113687] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7154.032990] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7154.063446] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7155.848579] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7161.436648] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 7165.430058] LustreError: 262020:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7165.476367] LustreError: 262020:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7170.529874] LustreError: 262021:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7172.772445] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7173.480300] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7180.172683] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 7188.402403] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7194.956106] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7202.417861] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7202.793591] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7210.397381] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 7217.029175] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7225.342060] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 7253.984608] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7253.990794] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7254.004771] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7256.035862] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7256.037075] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7256.040423] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7256.043470] Lustre: Skipped 1 previous similar message [ 7257.909767] Lustre: server umount lustre-OST0000 complete [ 7265.760699] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7265.767579] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7265.797410] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7269.065269] Lustre: server umount lustre-MDT0000 complete [ 7271.394120] LustreError: 262017:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7271.419175] LustreError: 262017:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7273.359869] LustreError: 262001:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790003091 with bad export cookie 12714133594941471800 [ 7273.367693] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7273.385861] LustreError: 262001:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7273.808014] Lustre: server umount lustre-MDT0001 complete [ 7287.027584] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 7290.441154] Key type lgssc unregistered [ 7290.731799] LNet: 266347:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7290.738169] LNetError: 266347:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7290.764133] LNet: Removed LNI 192.168.204.128@tcp [ 7291.853864] Key type .llcrypt unregistered [ 7291.859811] Key type ._llcrypt unregistered [ 7312.225605] Key type ._llcrypt registered [ 7312.227500] Key type .llcrypt registered [ 7312.324556] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 7313.116638] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7313.142758] alg: No test for adler32 (adler32-zlib) [ 7314.324477] Lustre: Lustre: Build Version: 2.17.58_63_g3df9749 [ 7314.554560] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 7316.191220] Key type lgssc registered [ 7317.748904] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7326.770823] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7326.793644] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7328.282465] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7333.215924] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 7338.474927] LustreError: 267584:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7338.496313] LustreError: 267584:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7343.291623] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7343.586281] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 7343.878823] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7351.188824] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 7362.058920] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7368.940623] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7376.847069] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7377.354987] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7385.075090] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 7393.324819] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7398.849306] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 7411.433868] Lustre: server umount lustre-OST0000 complete [ 7415.778798] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7415.787166] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7415.792714] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7419.364308] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7419.377329] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7424.488822] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7424.497575] Lustre: Skipped 2 previous similar messages [ 7429.601127] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7430.113381] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7430.496546] Lustre: server umount lustre-MDT0000 complete [ 7430.623167] Lustre: 266886:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790003232/real 1790003232] req@00000000477d0f3a x1876954324171776/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1790003248 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7430.626735] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7430.664615] Lustre: 266886:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 7430.682294] LustreError: 267584:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7430.723145] LustreError: 267584:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7434.890927] LustreError: 267566:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790003252 with bad export cookie 4912824776714684000 [ 7434.908608] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7434.909986] LustreError: 267566:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7435.377306] Lustre: server umount lustre-MDT0001 complete [ 7446.149682] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 7449.231561] Key type lgssc unregistered [ 7449.653482] LNet: 271266:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7449.662687] LNetError: 271266:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7449.676202] LNet: Removed LNI 192.168.204.128@tcp [ 7450.598097] Key type .llcrypt unregistered [ 7450.599947] Key type ._llcrypt unregistered [ 7477.736589] Key type ._llcrypt registered [ 7477.738876] Key type .llcrypt registered [ 7477.911399] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 7478.886694] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7478.992099] alg: No test for adler32 (adler32-zlib) [ 7480.014364] Lustre: Lustre: Build Version: 2.17.58_63_g3df9749 [ 7480.392284] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 7482.231294] Key type lgssc registered [ 7483.659201] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7496.282876] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7496.318502] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7498.115769] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7503.711799] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 7508.450519] LustreError: 272703:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7508.473565] LustreError: 272703:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7511.948794] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7512.224392] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7517.344262] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 7525.762905] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7532.032910] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7539.126769] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7539.526388] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7543.615527] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7545.500984] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 7554.403774] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7583.722702] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7583.728319] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7583.751953] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7583.768758] Lustre: Skipped 1 previous similar message [ 7588.549108] Lustre: server umount lustre-OST0000 complete [ 7593.952396] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7593.957173] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7593.964834] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7593.969909] Lustre: Skipped 2 previous similar messages [ 7593.983601] Lustre: Skipped 1 previous similar message [ 7598.565126] Lustre: server umount lustre-MDT0000 complete [ 7599.085182] LustreError: 272703:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7599.116934] LustreError: 272703:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7603.171640] LustreError: 273702:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7603.193976] LustreError: 273702:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7603.658335] LustreError: 272682:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790003421 with bad export cookie 7648753424405530188 [ 7603.663869] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7603.669783] LustreError: 272682:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7604.255590] Lustre: server umount lustre-MDT0001 complete [ 7617.156242] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 7619.778530] Key type lgssc unregistered [ 7620.078291] LNet: 277008:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7620.089500] LNetError: 277008:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7620.099566] LNet: Removed LNI 192.168.204.128@tcp [ 7620.860940] Key type .llcrypt unregistered [ 7620.863869] Key type ._llcrypt unregistered [ 7639.870453] Key type ._llcrypt registered [ 7639.875412] Key type .llcrypt registered [ 7640.006216] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 7641.363800] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7641.393508] alg: No test for adler32 (adler32-zlib) [ 7642.599130] Lustre: Lustre: Build Version: 2.17.58_63_g3df9749 [ 7642.953537] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 7644.688625] Key type lgssc registered [ 7646.571732] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7656.037342] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7656.057642] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7657.753315] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7664.489589] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 7667.171892] LustreError: 278262:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7667.205308] LustreError: 278262:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7672.289520] LustreError: 278264:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7675.125667] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7675.425549] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7679.690707] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 7688.982681] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7696.139495] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7704.033048] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7704.532097] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7711.656268] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 7720.410337] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7727.100226] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 7741.412377] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7741.430253] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7741.452188] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7742.442200] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7742.442454] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7742.459452] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7746.244150] Lustre: server umount lustre-OST0000 complete [ 7752.168398] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7752.179057] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7752.204024] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7752.206564] Lustre: Skipped 1 previous similar message [ 7756.965221] Lustre: server umount lustre-MDT0000 complete [ 7757.797786] LustreError: 278262:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7757.815335] LustreError: 278262:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7761.400817] LustreError: 280128:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790003579 with bad export cookie 17783200714581962656 [ 7761.405096] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7761.412958] LustreError: 280128:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7761.854891] Lustre: server umount lustre-MDT0001 complete [ 7774.585663] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 7777.949125] Key type lgssc unregistered [ 7778.309679] LNet: 281991:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7778.333506] LNetError: 281991:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7779.366967] LNet: Removed LNI 192.168.204.128@tcp [ 7780.516989] Key type .llcrypt unregistered [ 7780.523962] Key type ._llcrypt unregistered [ 7793.836156] Key type ._llcrypt registered [ 7793.837770] Key type .llcrypt registered [ 7793.968086] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 7811.411182] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_hostid [ 7827.764270] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 7829.235830] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7829.283213] alg: No test for adler32 (adler32-zlib) [ 7830.306795] Lustre: Lustre: Build Version: 2.17.58_63_g3df9749 [ 7830.573265] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 7832.271221] Key type lgssc registered [ 7834.421117] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7873.408417] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7873.481071] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7874.834744] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7874.885847] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7875.073551] Lustre: lustre-MDT0000: new disk, initializing [ 7875.239377] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7875.269698] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7880.350825] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 7892.328557] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7892.491659] Lustre: 285559:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7892.546279] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7892.550859] Lustre: Skipped 1 previous similar message [ 7892.666967] Lustre: lustre-MDT0001: new disk, initializing [ 7892.803145] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7892.851460] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7892.866594] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7898.066183] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 7909.742096] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7916.797240] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7925.384557] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7925.903952] Lustre: lustre-OST0000: new disk, initializing [ 7925.908013] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7925.913926] Lustre: 287526:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7925.987194] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7933.033574] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7933.068432] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7933.266139] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7934.980260] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 7945.488431] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7951.514603] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7951.920633] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7956.974347] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7957.290100] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7964.131035] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7964.138714] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7964.144259] LustreError: Skipped 1 previous similar message [ 7964.147575] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7964.175393] Lustre: Skipped 1 previous similar message [ 7965.687621] Lustre: server umount lustre-OST0000 complete [ 7969.771190] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7969.780711] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7969.799101] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7969.808977] Lustre: Skipped 1 previous similar message [ 7974.370500] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7974.371317] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7975.862842] Lustre: server umount lustre-MDT0000 complete [ 7980.049022] LustreError: 285552:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790003798 with bad export cookie 3769997135449199776 [ 7980.054778] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7980.062797] LustreError: 285552:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7980.537994] Lustre: server umount lustre-MDT0001 complete [ 7990.620810] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 11:16:45 (1790003805) [ 7998.352643] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7998.944146] LustreError: 289939:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7998.967851] LustreError: 289939:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 7999.061899] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8002.976254] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 8004.071333] LustreError: 289939:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8004.104082] LustreError: 289939:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8008.177312] LustreError: 289940:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8011.866178] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8012.490992] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8018.492175] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 8027.226303] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8035.173130] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8041.474587] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8042.062293] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8048.595315] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 8056.748962] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8068.577600] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8068.582244] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8068.594215] Lustre: Skipped 1 previous similar message [ 8068.610893] Lustre: Skipped 2 previous similar messages [ 8073.148490] Lustre: server umount lustre-OST0000 complete [ 8078.817804] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8078.825735] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8078.841654] Lustre: Skipped 1 previous similar message [ 8078.855594] Lustre: Skipped 1 previous similar message [ 8083.679304] Lustre: server umount lustre-MDT0000 complete [ 8083.936242] LustreError: 291375:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8083.966696] LustreError: 291375:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8087.837549] LustreError: 289920:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790003905 with bad export cookie 3769997135449200665 [ 8087.847380] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8087.852990] LustreError: 289920:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8088.383849] Lustre: server umount lustre-MDT0001 complete [ 8100.414721] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 8103.149052] Key type lgssc unregistered [ 8103.429747] LNet: 293475:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8103.444178] LNetError: 293475:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8103.472784] LNet: Removed LNI 192.168.204.128@tcp [ 8104.522729] Key type .llcrypt unregistered [ 8104.526314] Key type ._llcrypt unregistered [ 8134.322169] Key type ._llcrypt registered [ 8134.324571] Key type .llcrypt registered [ 8134.475885] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_hostid [ 8151.728901] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 8152.782977] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8152.856340] alg: No test for adler32 (adler32-zlib) [ 8154.059147] Lustre: Lustre: Build Version: 2.17.58_63_g3df9749 [ 8154.429584] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 8156.255261] Key type lgssc registered [ 8158.294241] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8212.897519] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8212.931177] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8214.392647] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8214.468244] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8214.603052] Lustre: lustre-MDT0000: new disk, initializing [ 8214.790990] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8214.837177] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8220.770238] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 8232.818784] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8232.998452] Lustre: 297335:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8233.055471] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8233.057916] Lustre: Skipped 1 previous similar message [ 8233.121335] Lustre: lustre-MDT0001: new disk, initializing [ 8233.220133] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8233.297936] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8233.306629] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8238.425055] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 8248.069150] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8255.394570] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8264.620630] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8265.004654] Lustre: lustre-OST2710: new disk, initializing [ 8265.011337] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 8265.023684] Lustre: 299305:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8265.179847] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 8271.470197] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 8273.580891] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 8273.598645] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 8273.720844] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 8281.671982] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8291.088737] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8291.307914] Lustre: lustre-OST03e8: new disk, initializing [ 8291.314788] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 8291.331077] Lustre: 300367:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8291.493141] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 8300.662606] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 8300.676742] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 8300.872820] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 8301.474096] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 8310.029238] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8318.760209] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 8319.111873] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8324.433473] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 8324.682615] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8330.035898] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 8330.311891] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8335.654486] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 8335.919474] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8340.961254] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8340.968857] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8340.990566] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8341.984254] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8341.986265] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8341.998437] Lustre: Skipped 1 previous similar message [ 8342.023504] Lustre: Skipped 1 previous similar message [ 8342.474726] Lustre: server umount lustre-MDT0000 complete [ 8346.485109] LustreError: 297329:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790004164 with bad export cookie 8133757528254625303 [ 8346.490057] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8346.505239] LustreError: 297329:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8346.958967] Lustre: server umount lustre-MDT0001 complete [ 8361.726207] Lustre: server umount lustre-OST2710 complete [ 8363.295132] Lustre: 295124:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790004165/real 1790004165] req@00000000df247d57 x1876955205015296/t0(0) o400->lustre-MDT0001-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790004181 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8363.327091] Lustre: lustre-MDT0001-lwp-OST03e8: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8363.337119] Lustre: Skipped 1 previous similar message [ 8365.680870] Lustre: server umount lustre-OST03e8 complete [ 8393.152617] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_hostid [ 8407.157615] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 8448.154675] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8448.360804] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8448.404303] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8448.501404] Lustre: lustre-MDT0000: new disk, initializing [ 8448.578747] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8448.591856] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8455.664568] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 8470.641866] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8470.721738] Lustre: 305642:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8470.774683] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8470.778191] Lustre: Skipped 1 previous similar message [ 8470.821352] Lustre: lustre-MDT0001: new disk, initializing [ 8470.982446] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8471.037580] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8471.067495] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8476.111705] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 8486.058093] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8492.215501] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8499.471637] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8499.838393] Lustre: lustre-OST0000: new disk, initializing [ 8499.844396] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8499.864649] Lustre: 307614:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8499.991454] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8505.910847] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8505.925756] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8506.040074] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8506.965373] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 8516.838446] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8521.816110] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8522.014460] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8526.420494] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8526.700145] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8531.944648] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8531.951785] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8531.954310] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8531.958341] Lustre: Skipped 1 previous similar message [ 8531.965079] Lustre: Skipped 3 previous similar messages [ 8534.880963] Lustre: server umount lustre-OST0000 complete [ 8542.179216] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8542.214361] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8542.228056] Lustre: Skipped 1 previous similar message [ 8545.191907] Lustre: server umount lustre-MDT0000 complete [ 8549.009823] LustreError: 305634:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790004366 with bad export cookie 8133757528254626983 [ 8549.019301] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8549.021371] LustreError: 305634:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8549.378877] Lustre: server umount lustre-MDT0001 complete [ 8558.142726] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 11:26:13 (1790004373) [ 8559.933969] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 8562.195959] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 11:26:17 (1790004377) [ 8567.798637] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 8602.419836] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8602.770734] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8602.801465] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8602.954249] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8607.905802] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 8617.684148] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8617.867758] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8617.937838] Lustre: 311472:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8623.901901] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 8646.896502] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8647.180160] Lustre: Denying initial registration attempt from nid 192.168.204.128@tcp, specified as failover [ 8647.190411] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 8647.205053] LustreError: 312927:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -99 [ 8647.217972] LustreError: 312927:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 8647.226811] LustreError: 312927:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 8647.329933] Lustre: server umount lustre-OST0000 complete [ 8647.334585] LustreError: 312927:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 8654.304354] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8654.310863] LustreError: Skipped 1 previous similar message [ 8654.319429] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8654.331547] Lustre: Skipped 1 previous similar message [ 8654.335055] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8660.310630] Lustre: server umount lustre-MDT0000 complete [ 8662.498770] LustreError: 311486:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8662.512477] LustreError: 311486:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 8664.427620] LustreError: 311466:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790004482 with bad export cookie 8133757528254627865 [ 8664.440900] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8664.450468] LustreError: 311466:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8664.913603] Lustre: server umount lustre-MDT0001 complete [ 8676.978464] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 8679.973535] Key type lgssc unregistered [ 8680.322897] LNet: 314048:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8680.328188] LNetError: 314048:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8680.340327] LNet: Removed LNI 192.168.204.128@tcp [ 8681.347674] Key type .llcrypt unregistered [ 8681.349656] Key type ._llcrypt unregistered [ 8693.801821] Key type ._llcrypt registered [ 8693.804333] Key type .llcrypt registered [ 8693.943979] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 8695.914192] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8695.995060] alg: No test for adler32 (adler32-zlib) [ 8697.230635] Lustre: Lustre: Build Version: 2.17.58_63_g3df9749 [ 8697.592908] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 8699.384313] Key type lgssc registered [ 8700.817808] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8750.637702] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 8760.752966] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8760.776459] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8762.043260] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8762.070605] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8762.263312] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8767.198753] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 8775.391450] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8775.468308] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8775.538896] Lustre: 316828:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8775.782762] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8780.734095] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 8787.953602] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8793.169961] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8812.596951] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8812.923919] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8813.325037] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8821.329274] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 8831.207306] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8838.115892] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8838.118705] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8838.153073] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8838.174281] Lustre: Skipped 1 previous similar message [ 8839.737216] Lustre: server umount lustre-OST0000 complete [ 8847.335743] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8847.346181] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8847.373930] Lustre: Skipped 1 previous similar message [ 8847.390776] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8848.353879] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8850.500252] Lustre: server umount lustre-MDT0000 complete [ 8854.169907] LustreError: 316822:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790004672 with bad export cookie 4019466295504101745 [ 8854.184901] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8854.188161] LustreError: 316822:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8854.665419] Lustre: server umount lustre-MDT0001 complete [ 8867.132664] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 8870.534799] Key type lgssc unregistered [ 8870.901030] LNet: 320408:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8870.911446] LNetError: 320408:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8870.938501] LNet: Removed LNI 192.168.204.128@tcp [ 8872.214070] Key type .llcrypt unregistered [ 8872.218565] Key type ._llcrypt unregistered [ 8900.617885] Key type ._llcrypt registered [ 8900.620515] Key type .llcrypt registered [ 8900.695749] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 8902.035042] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8902.080622] alg: No test for adler32 (adler32-zlib) [ 8903.255318] Lustre: Lustre: Build Version: 2.17.58_63_g3df9749 [ 8903.533219] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 8905.179271] Key type lgssc registered [ 8906.733652] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8915.680743] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8915.694287] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8917.263362] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8921.896257] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 8927.714537] LustreError: 321843:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8927.735829] LustreError: 321843:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8930.512052] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8930.870229] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8935.928440] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 8944.889664] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8951.875062] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8961.060905] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8961.327560] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8967.979673] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 8975.733670] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8987.110164] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8987.110548] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8987.116527] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8987.126039] Lustre: Skipped 1 previous similar message [ 8988.801787] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.28@tcp (stopping) [ 8988.807870] Lustre: Skipped 2 previous similar messages [ 8990.829141] Lustre: server umount lustre-MDT0000 complete [ 8992.228726] LustreError: 321839:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8992.242346] LustreError: 321839:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8993.943077] LustreError: 321840:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8995.038786] LustreError: 321823:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790004813 with bad export cookie 10405852603267411678 [ 8995.042691] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8995.049679] LustreError: 321823:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8997.345185] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8997.347669] LustreError: 321838:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8997.356167] Lustre: Skipped 1 previous similar message [ 8997.358474] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9000.368470] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9000.391481] Lustre: Skipped 1 previous similar message [ 9001.455482] Lustre: server umount lustre-MDT0001 complete [ 9009.257115] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9018.668330] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9020.832252] LustreError: 325424:0:(import.c:339:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [ 9020.849943] LustreError: 325424:0:(import.c:363:ptlrpc_invalidate_import()) @@@ still on sending list req@00000000bbfff038 x1876955990372096/t0(0) o250->MGC192.168.204.128@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1790004838 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9020.882678] LustreError: 325424:0:(import.c:373:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. [ 9021.313339] LustreError: 323676:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9021.336776] LustreError: 323676:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 9021.388729] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9021.463240] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 9026.528796] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 9030.630685] LustreError: 325435:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9030.633497] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9030.653971] LustreError: 325435:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 9033.714575] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9034.053657] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9037.502484] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 9039.348850] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9045.712805] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9047.624608] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9053.985410] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9056.879458] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9062.370611] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9062.386635] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9062.399311] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9064.937514] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9064.947913] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9068.370851] Lustre: server umount lustre-OST0000 complete [ 9075.169341] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9075.171106] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9075.171511] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9075.171515] Lustre: Skipped 2 previous similar messages [ 9075.179488] Lustre: Skipped 1 previous similar message [ 9077.911137] Lustre: server umount lustre-MDT0000 complete [ 9080.290356] LustreError: 325441:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9080.312455] LustreError: 325441:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 9081.639774] LustreError: 324428:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790004899 with bad export cookie 10405852603267419518 [ 9081.641683] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9081.647941] LustreError: 324428:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9081.997605] Lustre: server umount lustre-MDT0001 complete [ 9094.854696] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 9098.237334] Key type lgssc unregistered [ 9098.531292] LNet: 328111:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9098.537193] LNetError: 328111:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9098.573733] LNet: Removed LNI 192.168.204.128@tcp [ 9099.583874] Key type .llcrypt unregistered [ 9099.586594] Key type ._llcrypt unregistered [ 9128.508614] Key type ._llcrypt registered [ 9128.514879] Key type .llcrypt registered [ 9128.700325] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 9130.350506] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9130.403378] alg: No test for adler32 (adler32-zlib) [ 9131.633284] Lustre: Lustre: Build Version: 2.17.58_63_g3df9749 [ 9132.108506] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 9133.919264] Key type lgssc registered [ 9135.510827] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9144.152952] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9144.207103] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9145.880044] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9150.352341] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 9155.045884] LustreError: 329544:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9155.069697] LustreError: 329544:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 9158.647271] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9159.182940] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9164.492905] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 9173.550925] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9179.759929] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9184.749827] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9184.756232] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9184.767795] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9185.765714] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9185.768272] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9186.849972] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9186.855447] Lustre: Skipped 2 previous similar messages [ 9189.190677] Lustre: server umount lustre-MDT0000 complete [ 9191.905184] LustreError: 329544:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9191.942679] LustreError: 329544:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9193.354880] LustreError: 329525:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790005011 with bad export cookie 4572411990490739188 [ 9193.358821] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9193.365903] LustreError: 329525:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9193.922404] Lustre: server umount lustre-MDT0001 complete [ 9209.220591] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9209.690804] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9209.746036] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9210.069568] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9216.075309] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 9225.429111] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9225.532198] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9225.582805] Lustre: 332131:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9225.959287] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9231.142041] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 9239.799904] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9245.656528] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9256.047455] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9256.342479] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 9256.358766] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 9256.371378] LustreError: 334019:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -2 [ 9256.376519] LustreError: 334019:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 9256.383806] LustreError: 334019:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9256.496272] Lustre: server umount lustre-OST0000 complete [ 9256.503088] LustreError: 334019:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 9267.890103] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9268.121723] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9268.268775] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9274.910561] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 9277.641333] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 9284.008975] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9292.669852] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9292.790694] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 9292.868902] Lustre: lustre-OST0001: new disk, initializing [ 9292.872596] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9292.878882] Lustre: 335557:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9293.022699] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9299.066034] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9299.084940] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9299.224124] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9300.686820] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 9309.882571] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9314.787637] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9314.795590] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9314.817807] Lustre: Skipped 1 previous similar message [ 9314.821364] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9314.829768] Lustre: Skipped 1 previous similar message [ 9319.626783] Lustre: server umount lustre-OST0001 complete [ 9319.922563] LustreError: 334486:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9325.041407] LustreError: 335849:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9325.046185] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9325.079405] LustreError: 335849:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 9325.116949] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9325.144632] Lustre: Skipped 1 previous similar message [ 9330.151123] LustreError: 334487:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9330.182640] LustreError: 334487:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9335.273205] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9335.279462] Lustre: Skipped 3 previous similar messages [ 9338.339168] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9338.627563] Lustre: server umount lustre-OST0000 complete [ 9343.970704] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9343.991715] LustreError: Skipped 1 previous similar message [ 9344.002531] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9344.023062] Lustre: Skipped 1 previous similar message [ 9355.745490] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9355.756789] Lustre: Skipped 6 previous similar messages [ 9357.279205] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9357.484296] Lustre: server umount lustre-MDT0000 complete [ 9360.877918] LustreError: 332142:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9360.909812] LustreError: 332142:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 9362.456390] LustreError: 332122:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790005180 with bad export cookie 4572411990490739741 [ 9362.462721] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9362.469541] LustreError: 332122:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9363.031159] Lustre: server umount lustre-MDT0001 complete [ 9377.034640] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 9380.093177] Key type lgssc unregistered [ 9380.405778] LNet: 337519:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9380.430061] LNetError: 337519:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9380.461923] LNet: Removed LNI 192.168.204.128@tcp [ 9381.230944] Key type .llcrypt unregistered [ 9381.233932] Key type ._llcrypt unregistered [ 9454.146221] Key type ._llcrypt registered [ 9454.148214] Key type .llcrypt registered [ 9454.249712] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_hostid [ 9469.225695] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 9470.019983] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9470.066688] alg: No test for adler32 (adler32-zlib) [ 9471.117354] Lustre: Lustre: Build Version: 2.17.58_63_g3df9749 [ 9471.493567] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 9473.223207] Key type lgssc registered [ 9475.575564] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9518.783062] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9518.824198] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9520.391115] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9520.426283] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9520.664971] Lustre: lustre-MDT0000: new disk, initializing [ 9520.753925] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9520.790121] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9526.912403] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 9541.583832] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9541.710583] Lustre: 342467:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9541.766208] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9541.773723] Lustre: Skipped 1 previous similar message [ 9541.838201] Lustre: lustre-MDT0001: new disk, initializing [ 9541.949854] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9541.999952] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9542.013859] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9546.771591] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 9555.412931] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9561.809234] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9571.805804] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9572.234293] Lustre: lustre-OST0000: new disk, initializing [ 9572.246083] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9572.254256] Lustre: 344439:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9572.379329] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9577.584988] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9577.600227] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9577.704863] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9579.505838] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 9589.334743] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9595.721809] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9595.993943] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9601.769297] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9601.975787] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9608.677868] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9608.686740] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9608.699333] LustreError: Skipped 1 previous similar message [ 9608.711163] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9608.774158] Lustre: Skipped 1 previous similar message [ 9610.923828] Lustre: server umount lustre-OST0000 complete [ 9618.918426] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9618.920287] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9618.921631] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9618.921637] Lustre: Skipped 1 previous similar message [ 9618.964674] Lustre: Skipped 1 previous similar message [ 9622.841074] Lustre: server umount lustre-MDT0000 complete [ 9626.117113] LustreError: 343265:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790005444 with bad export cookie 9801128740143709546 [ 9626.131702] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9626.148724] LustreError: 343265:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9626.538674] Lustre: server umount lustre-MDT0001 complete [ 9634.599881] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 11:44:09 (1790005449) [ 9641.164786] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 11:44:16 (1790005456) [ 9649.845460] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9650.290342] LustreError: 347195:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9650.341119] LustreError: 347195:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 9650.396570] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9655.789533] LustreError: 347196:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9656.291924] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 9660.900840] LustreError: 347195:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9665.914178] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9666.030784] LustreError: 347195:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9666.666264] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9671.028557] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 9680.102540] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9690.157866] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9699.045605] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9699.590634] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9707.038851] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 9716.095378] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9728.725922] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 9730.397974] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 9731.994218] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 9733.848935] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 9735.783529] Lustre: Failing over lustre-MDT0000 [ 9736.146740] Lustre: server umount lustre-MDT0000 complete [ 9737.413528] LustreError: 347192:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9738.209203] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9738.219873] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9740.769512] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9740.803691] Lustre: Skipped 1 previous similar message [ 9745.890509] LustreError: 347196:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9745.917168] LustreError: 347196:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [ 9756.126613] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9756.233935] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9756.552221] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9756.595719] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9758.835856] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9761.782844] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9761.813930] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 9761.839418] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 9761.867256] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 9771.795224] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9773.146719] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9777.434916] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 9779.676691] Lustre: Failing over lustre-MDT0000 [ 9780.192776] Lustre: server umount lustre-MDT0000 complete [ 9782.249159] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9782.253668] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9782.254896] LustreError: 347190:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9782.268719] Lustre: Skipped 1 previous similar message [ 9782.299878] LustreError: 347190:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 11 previous similar messages [ 9798.624197] Lustre: 340684:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790005600/real 1790005600] req@00000000d6f5f683 x1876956586014464/t0(0) o400->MGC192.168.204.128@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1790005616 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9798.662112] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9801.761825] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9808.874824] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x88049aa91a2347ff [ 9808.888112] Lustre: MGC192.168.204.128@tcp: Connection restored to 0@lo (at 0@lo) [ 9808.900616] Lustre: Skipped 2 previous similar messages [ 9809.084947] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9809.158949] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9811.061495] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9814.505785] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9814.549646] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 9814.639639] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 9814.790244] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 9826.708955] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9828.387442] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9840.097219] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9840.099932] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9840.108541] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9840.119307] Lustre: Skipped 2 previous similar messages [ 9840.132511] Lustre: Skipped 3 previous similar messages [ 9841.630852] Lustre: server umount lustre-MDT0000 complete [ 9845.218272] LustreError: 347195:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9845.251194] LustreError: 347195:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 26 previous similar messages [ 9846.035393] LustreError: 347177:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790005664 with bad export cookie 9801128740143712255 [ 9846.044532] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9846.051734] LustreError: 347177:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9846.447334] Lustre: server umount lustre-MDT0001 complete [ 9860.372148] Lustre: server umount lustre-OST0000 complete [ 9878.331476] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9878.997491] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9884.845411] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 9895.218473] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9902.213095] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 9912.399278] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9921.131034] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9930.628409] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9939.316244] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 9947.690424] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9948.494347] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 9958.894768] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9958.910235] Lustre: Skipped 1 previous similar message [ 9958.910843] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9958.913492] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9964.016971] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9964.030288] Lustre: Skipped 2 previous similar messages [ 9964.799767] Lustre: server umount lustre-OST0000 complete [ 9972.705520] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9976.672698] Lustre: server umount lustre-MDT0000 complete [ 9979.362373] LustreError: 353562:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9979.384672] LustreError: 353562:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 9981.018256] LustreError: 353548:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790005798 with bad export cookie 9801128740143713004 [ 9981.033610] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9981.034470] LustreError: 353548:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9981.398111] Lustre: server umount lustre-MDT0001 complete [ 9997.975870] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [10001.955568] Key type lgssc unregistered [10002.404597] LNet: 357102:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10002.418727] LNetError: 357102:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10002.432734] LNet: Removed LNI 192.168.204.128@tcp [10003.802078] Key type .llcrypt unregistered [10003.805412] Key type ._llcrypt unregistered [10041.525128] Key type ._llcrypt registered [10041.529503] Key type .llcrypt registered [10041.716760] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [10043.993865] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10044.007508] alg: No test for adler32 (adler32-zlib) [10045.101892] Lustre: Lustre: Build Version: 2.17.58_63_g3df9749 [10045.493069] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [10047.240623] Key type lgssc registered [10049.160425] Lustre: Echo OBD driver; http://www.lustre.org/ [10058.523948] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10058.547454] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10060.135687] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10066.714107] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [10070.513454] LustreError: 358643:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10070.553753] LustreError: 358643:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [10071.526922] LustreError: 358642:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10076.651075] LustreError: 358643:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10077.539886] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10078.070481] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10084.383292] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [10093.307038] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10099.682991] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10108.654219] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10109.371798] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10117.314504] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [10127.871646] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10131.902071] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [10147.300328] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10147.307379] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10147.323303] Lustre: Skipped 2 previous similar messages [10147.349315] Lustre: Skipped 2 previous similar messages [10151.767223] Lustre: server umount lustre-MDT0000 complete [10152.418318] LustreError: 359371:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10152.452343] LustreError: 359371:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [10156.816240] LustreError: 358625:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790005974 with bad export cookie 4866265779434847095 [10156.821219] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10156.831195] LustreError: 358625:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10157.161155] Lustre: server umount lustre-MDT0001 complete [10172.705304] Lustre: server umount lustre-OST0000 complete [10189.316653] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10190.105865] LustreError: 362373:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10190.224351] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10195.929055] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [10200.548795] LustreError: 362374:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10200.581821] LustreError: 362374:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10206.808716] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10207.416391] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10213.538827] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [10222.249703] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10228.518603] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10236.682324] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10237.245585] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10245.320520] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [10255.065773] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10260.805112] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [10276.321853] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10276.332852] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10276.357337] Lustre: Skipped 2 previous similar messages [10276.377053] Lustre: Skipped 2 previous similar messages [10281.006853] Lustre: server umount lustre-MDT0000 complete [10281.446083] LustreError: 362374:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10281.464301] LustreError: 362374:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10285.665780] LustreError: 362356:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790006103 with bad export cookie 4866265779434848208 [10285.673285] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10285.679462] LustreError: 362356:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10286.284869] Lustre: server umount lustre-MDT0001 complete [10301.690539] Lustre: server umount lustre-OST0000 complete [10321.140940] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10321.815774] LustreError: 366287:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10321.839375] LustreError: 366287:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [10321.922381] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10328.096851] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [10338.278701] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10343.874929] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [10354.285623] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10360.253445] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10369.227130] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10370.100056] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10370.113322] Lustre: Skipped 1 previous similar message [10382.824444] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [10391.598139] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [10394.532689] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10399.712256] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10399.717081] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10399.727134] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10401.769199] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10401.772227] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10401.780699] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10401.807399] Lustre: Skipped 1 previous similar message [10405.527954] Lustre: server umount lustre-OST0000 complete [10410.471882] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10410.481618] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10410.499495] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10416.393530] Lustre: server umount lustre-MDT0000 complete [10420.195536] LustreError: 367275:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10420.237127] LustreError: 367275:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [10420.476463] LustreError: 367024:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790006238 with bad export cookie 4866265779434849230 [10420.478726] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10420.501301] LustreError: 367024:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10420.997742] Lustre: server umount lustre-MDT0001 complete [10433.647672] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [10436.563405] Key type lgssc unregistered [10436.844421] LNet: 369825:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10436.856070] LNetError: 369825:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10437.926320] LNet: Removed LNI 192.168.204.128@tcp [10438.754727] Key type .llcrypt unregistered [10438.763128] Key type ._llcrypt unregistered [10472.972705] Key type ._llcrypt registered [10472.978744] Key type .llcrypt registered [10473.181437] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [10474.868610] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10475.144366] alg: No test for adler32 (adler32-zlib) [10476.921790] Lustre: Lustre: Build Version: 2.17.58_63_g3df9749 [10477.295966] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [10478.999579] Key type lgssc registered [10480.957580] Lustre: Echo OBD driver; http://www.lustre.org/ [10490.464871] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10490.480773] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10490.492617] LustreError: 371285:0:(osd_handler.c:8546:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [10490.528932] LustreError: 371285:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [10490.538164] LustreError: 371285:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [10490.547708] LustreError: 371285:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [10490.561215] LustreError: 371285:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [10499.740948] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [10499.747753] LustreError: 371682:0:(osd_handler.c:8546:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [10499.768238] LustreError: 371682:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [10499.779776] LustreError: 371682:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [10499.791966] LustreError: 371682:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [10499.803447] LustreError: 371682:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [10523.053272] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [10526.251929] Key type lgssc unregistered [10526.642960] LNet: 372684:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10526.653318] LNetError: 372684:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10526.685544] LNet: Removed LNI 192.168.204.128@tcp [10527.844584] Key type .llcrypt unregistered [10527.852039] Key type ._llcrypt unregistered [10548.981825] Key type ._llcrypt registered [10548.984166] Key type .llcrypt registered [10549.135213] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_hostid [10571.993218] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [10574.568797] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10574.695487] alg: No test for adler32 (adler32-zlib) [10576.126603] Lustre: Lustre: Build Version: 2.17.58_63_g3df9749 [10576.456590] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [10578.143757] Key type lgssc registered [10579.387349] Lustre: Echo OBD driver; http://www.lustre.org/ [10620.138067] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10620.176707] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10621.569854] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10621.640640] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10621.766607] Lustre: lustre-MDT0000: new disk, initializing [10621.935370] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10621.967566] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10626.734972] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [10637.006058] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10637.167251] Lustre: 375860:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10637.240636] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10637.244309] Lustre: Skipped 1 previous similar message [10637.292587] Lustre: lustre-MDT0001: new disk, initializing [10637.366840] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10637.393674] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10637.402131] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10643.136029] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [10655.457385] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10662.269473] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10668.623197] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10668.890745] Lustre: lustre-OST0000: new disk, initializing [10668.895334] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10668.908661] Lustre: 377841:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10669.096765] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10675.421592] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [10676.363042] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10676.379357] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10676.537118] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10685.810087] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10691.667293] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10691.913783] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10696.834575] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10697.077285] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10702.304337] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10702.308498] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10702.313697] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10702.323682] Lustre: Skipped 1 previous similar message [10707.442283] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10707.454163] Lustre: Skipped 2 previous similar messages [10712.552432] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10712.564077] Lustre: Skipped 1 previous similar message [10713.055846] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10713.401347] Lustre: server umount lustre-OST0000 complete [10717.667419] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10717.669297] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10717.682943] Lustre: Skipped 1 previous similar message [10717.699403] Lustre: Skipped 1 previous similar message [10722.784567] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10722.794586] Lustre: Skipped 1 previous similar message [10723.474407] Lustre: server umount lustre-MDT0000 complete [10727.219491] LustreError: 375850:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790006545 with bad export cookie 3587335663310355505 [10727.224228] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10727.235836] LustreError: 375850:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10727.566763] Lustre: server umount lustre-MDT0001 complete [10735.551644] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [10737.043379] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 12:02:32 (1790006552) [10743.482231] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10744.132965] LustreError: 380302:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10744.174211] LustreError: 380302:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [10744.315970] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10749.417136] LustreError: 380303:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10752.266349] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [10754.529065] LustreError: 380302:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10759.649994] LustreError: 380303:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10764.797610] LustreError: 380302:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10766.434148] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10766.987690] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10771.130157] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [10777.558568] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10784.237833] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10791.963299] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10792.421317] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10798.684132] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [10806.372913] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10812.152703] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10812.379606] Lustre: lustre-OST0001: new disk, initializing [10812.384887] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10812.392597] Lustre: 383138:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10812.523868] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10818.760128] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10818.781685] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10818.929975] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10819.175624] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [10830.583275] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10834.405475] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10834.408074] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10834.420399] LustreError: Skipped 1 previous similar message [10834.421877] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10834.484147] Lustre: Skipped 1 previous similar message [10840.639730] Lustre: server umount lustre-OST0001 complete [10842.228936] LustreError: 382127:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10857.958479] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10857.971512] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10857.994401] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10857.999664] Lustre: Skipped 4 previous similar messages [10860.006519] LustreError: 382126:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10860.022988] LustreError: 382126:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [10871.263733] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10871.434609] Lustre: server umount lustre-OST0000 complete [10878.435585] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10878.452535] Lustre: Skipped 2 previous similar messages [10882.431474] Lustre: server umount lustre-MDT0000 complete [10888.254419] LustreError: 380283:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790006706 with bad export cookie 3587335663310356387 [10888.268257] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10888.281236] LustreError: 380283:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10894.916996] Lustre: server umount lustre-MDT0001 complete [10908.151192] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [10911.225609] Key type lgssc unregistered [10911.643900] LNet: 385127:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10911.650706] LNetError: 385127:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10911.693844] LNet: Removed LNI 192.168.204.128@tcp [10912.835783] Key type .llcrypt unregistered [10912.837961] Key type ._llcrypt unregistered [10971.631451] Key type ._llcrypt registered [10971.635877] Key type .llcrypt registered [10971.775694] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [10972.795539] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10972.895404] alg: No test for adler32 (adler32-zlib) [10973.907516] Lustre: Lustre: Build Version: 2.17.58_63_g3df9749 [10974.088035] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [10975.807329] Key type lgssc registered [10977.277748] Lustre: Echo OBD driver; http://www.lustre.org/ [10985.847655] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10985.861847] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10987.199177] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10987.264275] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10987.533923] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10993.346924] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [11004.371320] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11004.437554] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11004.468849] Lustre: 387943:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11004.577550] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [11004.774717] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11010.173394] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [11017.521628] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11023.211647] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11030.497100] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11030.499296] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11030.518050] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11030.551040] Lustre: Skipped 1 previous similar message [11031.953713] Lustre: server umount lustre-MDT0000 complete [11036.364917] LustreError: 388686:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790006854 with bad export cookie 13005696183249093694 [11036.369695] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11036.379842] LustreError: 388686:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11036.749304] Lustre: server umount lustre-MDT0001 complete [11046.444509] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 12:07:41 (1790006861) [11054.654192] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11055.501607] LustreError: 390457:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11055.541335] LustreError: 390457:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [11055.605489] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11060.721170] LustreError: 390458:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11062.202644] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [11065.835225] LustreError: 390457:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11070.954960] LustreError: 390458:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11072.750385] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11073.158718] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11078.226826] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [11085.910737] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11092.614265] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11098.463848] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11098.710321] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [11098.911691] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11105.291685] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [11115.775389] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11124.616202] Lustre: Permanently deactivating lustre-OST0000 [11124.624573] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [11132.037715] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11133.987391] LustreError: 393398:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [11134.010995] LustreError: 393398:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11134.027193] LustreError: 393398:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11136.644781] LustreError: 393446:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11136.656835] LustreError: 393446:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11145.282313] Lustre: server umount lustre-OST0000 complete [11148.449847] LustreError: 393696:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11148.454627] LustreError: 393696:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11154.912402] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11154.918225] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11154.927157] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11154.941652] Lustre: Skipped 1 previous similar message [11154.950520] Lustre: Skipped 1 previous similar message [11160.043170] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11160.049122] Lustre: Skipped 2 previous similar messages [11161.122856] Lustre: server umount lustre-MDT0000 complete [11165.154358] LustreError: 390457:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11165.176852] LustreError: 390457:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11165.384585] LustreError: 390439:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790006983 with bad export cookie 13005696183249094275 [11165.390650] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11165.398071] LustreError: 390439:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11176.149577] Lustre: server umount lustre-MDT0001 complete [11186.146658] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11190.876969] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [11193.726521] LustreError: 394857:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [11200.990184] Lustre: 395196:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.128@tcp [11204.259039] Lustre: server umount MGS complete [11218.386173] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11219.106080] LustreError: 395922:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11219.142685] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11219.151582] Lustre: Skipped 1 previous similar message [11219.235256] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11224.530838] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [11236.055245] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11236.468295] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11236.573308] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11242.632385] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [11250.645087] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11256.044744] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11257.838421] Lustre: Permanently reactivating lustre-OST0000 [11257.863769] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [11257.870872] Lustre: Skipped 2 previous similar messages [11287.519168] LustreError: 397714:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [11297.983232] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11299.088387] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11309.269564] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [11317.780155] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11327.970403] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11327.971364] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11327.972213] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11328.004860] Lustre: Skipped 1 previous similar message [11330.703343] Lustre: server umount lustre-OST0000 complete [11338.211874] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11338.215988] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11338.234363] Lustre: Skipped 1 previous similar message [11338.255300] Lustre: Skipped 2 previous similar messages [11341.820841] Lustre: server umount lustre-MDT0000 complete [11343.328885] LustreError: 395922:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11343.342729] LustreError: 395922:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [11346.415223] LustreError: 395903:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790007164 with bad export cookie 13005696183249095325 [11346.427227] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11346.432369] LustreError: 395903:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11346.846674] Lustre: server umount lustre-MDT0001 complete [11360.223145] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [11364.037681] Key type lgssc unregistered [11364.454703] LNet: 400292:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11364.461516] LNetError: 400292:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11364.477792] LNet: Removed LNI 192.168.204.128@tcp [11365.475188] Key type .llcrypt unregistered [11365.477344] Key type ._llcrypt unregistered [11387.739770] Key type ._llcrypt registered [11387.742734] Key type .llcrypt registered [11387.904416] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_hostid [11404.905404] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [11405.827227] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11405.845449] alg: No test for adler32 (adler32-zlib) [11407.124652] Lustre: Lustre: Build Version: 2.17.58_63_g3df9749 [11407.459191] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [11409.208813] Key type lgssc registered [11410.767897] Lustre: Echo OBD driver; http://www.lustre.org/ [11448.912638] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11448.949453] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11450.214234] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11450.251627] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11450.394663] Lustre: lustre-MDT0000: new disk, initializing [11450.514138] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11450.546060] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11455.840711] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [11468.254316] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11468.435625] Lustre: 403467:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11468.522271] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11468.534546] Lustre: Skipped 1 previous similar message [11468.663321] Lustre: lustre-MDT0001: new disk, initializing [11468.833198] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11468.888702] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11468.900567] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11474.480584] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [11485.927568] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11491.807203] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11500.624833] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11501.130552] Lustre: lustre-OST0000: new disk, initializing [11501.142689] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11501.159178] Lustre: 405434:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11501.336661] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11508.396471] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [11511.386100] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11511.396180] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11511.527134] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11520.127917] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11527.205925] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11527.594354] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11534.442927] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11534.705936] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11537.380746] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11537.389515] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11537.395510] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11537.409949] Lustre: Skipped 1 previous similar message [11542.510657] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11542.513476] Lustre: Skipped 2 previous similar messages [11543.389078] Lustre: server umount lustre-OST0000 complete [11550.688596] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11550.726497] LustreError: Skipped 1 previous similar message [11550.729146] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11550.750743] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11552.757590] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11552.774941] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11552.786349] Lustre: Skipped 1 previous similar message [11554.104183] Lustre: server umount lustre-MDT0000 complete [11557.716605] LustreError: 403459:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790007375 with bad export cookie 11110577433047375427 [11557.717114] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11557.728218] LustreError: 403459:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11558.132048] Lustre: server umount lustre-MDT0001 complete [11567.455655] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [11569.407156] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 12:16:24 (1790007384) [11577.338893] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11577.933086] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11582.953304] LustreError: 407895:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11582.975287] LustreError: 407895:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11583.173870] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [11587.047202] LustreError: 407896:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11592.166815] LustreError: 407895:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11592.532740] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11593.147503] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11598.617155] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [11606.765248] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11612.326632] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11620.107722] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11620.575198] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11627.746817] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [11635.643880] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11637.717138] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [11640.612497] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [11648.480556] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11648.501458] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11648.546763] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11652.605968] Lustre: server umount lustre-OST0000 complete [11659.745918] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11659.747124] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11659.749576] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11659.749585] Lustre: Skipped 2 previous similar messages [11659.760346] Lustre: Skipped 2 previous similar messages [11659.803830] LustreError: Skipped 1 previous similar message [11663.390236] Lustre: server umount lustre-MDT0000 complete [11664.866647] LustreError: 407896:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11667.589200] LustreError: 407877:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790007485 with bad export cookie 11110577433047376309 [11667.598068] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11667.601476] LustreError: 407877:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11668.245146] Lustre: server umount lustre-MDT0001 complete [11681.214743] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [11684.057459] Key type lgssc unregistered [11684.312647] LNet: 411623:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11684.319077] LNetError: 411623:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11684.331105] LNet: Removed LNI 192.168.204.128@tcp [11685.210716] Key type .llcrypt unregistered [11685.214165] Key type ._llcrypt unregistered [11714.755727] Key type ._llcrypt registered [11714.759194] Key type .llcrypt registered [11714.963754] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [11715.322292] Key type .llcrypt unregistered [11715.324354] Key type ._llcrypt unregistered [11733.029772] Key type ._llcrypt registered [11733.038187] Key type .llcrypt registered [11733.180990] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [11733.838939] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11733.884154] alg: No test for adler32 (adler32-zlib) [11735.018682] Lustre: Lustre: Build Version: 2.17.58_63_g3df9749 [11735.362033] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [11737.151644] Key type lgssc registered [11738.901290] Lustre: Echo OBD driver; http://www.lustre.org/ [11749.498707] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11749.514534] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11750.984442] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11755.745397] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [11760.097548] LustreError: 413849:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11760.121564] LustreError: 413849:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11765.219257] LustreError: 413850:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11765.397438] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11765.810361] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11770.853748] LustreError: 413849:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11770.882395] LustreError: 413849:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11773.029884] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [11775.973532] LustreError: 413850:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11776.018107] LustreError: 413850:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11776.025215] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [11781.094311] LustreError: 414840:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11781.101303] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11781.120218] LustreError: 414840:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11787.836905] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11788.261600] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11793.499236] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [11803.627455] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11803.628664] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11803.661159] Lustre: Skipped 1 previous similar message [11808.545280] Lustre: server umount lustre-OST0000 complete [11813.857442] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11813.858668] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11813.881466] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11813.888845] Lustre: Skipped 2 previous similar messages [11818.720595] Lustre: server umount lustre-MDT0000 complete [11818.978232] LustreError: 413849:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11818.998548] LustreError: 413849:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [11822.568628] LustreError: 413832:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790007640 with bad export cookie 1118104007274704638 [11822.575848] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11822.581778] LustreError: 413832:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11823.154987] Lustre: server umount lustre-MDT0001 complete [11838.580846] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [11842.493644] Key type lgssc unregistered [11842.947291] LNet: 416999:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11842.968536] LNetError: 416999:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11844.007240] LNet: Removed LNI 192.168.204.128@tcp [11844.949888] Key type .llcrypt unregistered [11844.953677] Key type ._llcrypt unregistered [11870.500992] Key type ._llcrypt registered [11870.505457] Key type .llcrypt registered [11870.622791] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [11871.589791] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11871.641898] alg: No test for adler32 (adler32-zlib) [11872.743420] Lustre: Lustre: Build Version: 2.17.58_63_g3df9749 [11873.029778] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [11874.807220] Key type lgssc registered [11876.446139] Lustre: Echo OBD driver; http://www.lustre.org/ [11886.371519] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11886.402882] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11901.983465] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [11907.104890] LustreError: 418414:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.128@tcp: failed processing log, type 4: rc = -110 [11933.793563] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11940.781774] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [11948.721571] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11956.348481] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11956.880168] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11956.957380] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [11962.345783] LustreError: 418440:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11962.388588] LustreError: 418440:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11962.801458] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [11967.458033] LustreError: 418439:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11968.492268] LustreError: 419454:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11968.525946] LustreError: 419454:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11972.474276] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11972.910379] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11978.387394] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [11986.961618] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11992.742733] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12002.547644] Lustre: server umount lustre-OST0000 complete [12007.917352] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [12007.933848] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12007.952877] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12008.929391] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12008.930934] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12008.946786] Lustre: Skipped 1 previous similar message [12014.051338] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12014.064425] Lustre: Skipped 1 previous similar message [12016.168877] Lustre: server umount lustre-MDT0000 complete [12019.167266] Lustre: 417744:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790007821/real 1790007821] req@00000000a21f8aff x1876959103580672/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1790007837 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12019.168575] LustreError: 419454:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12019.170328] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12019.202837] Lustre: 417744:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [12019.239831] LustreError: 419454:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12020.586745] LustreError: 420943:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790007838 with bad export cookie 12701501913531445707 [12020.593568] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12020.595343] LustreError: 420943:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12020.833138] Lustre: server umount lustre-MDT0001 complete [12033.567922] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [12036.865645] Key type lgssc unregistered [12037.230270] LNet: 421987:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12037.237237] LNetError: 421987:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12037.276830] LNet: Removed LNI 192.168.204.128@tcp [12038.466554] Key type .llcrypt unregistered [12038.469669] Key type ._llcrypt unregistered [12067.321484] Key type ._llcrypt registered [12067.323616] Key type .llcrypt registered [12067.405353] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [12068.752990] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12068.799122] alg: No test for adler32 (adler32-zlib) [12069.928667] Lustre: Lustre: Build Version: 2.17.58_63_g3df9749 [12070.274641] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [12071.991254] Key type lgssc registered [12073.414772] Lustre: Echo OBD driver; http://www.lustre.org/ [12082.422544] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12082.456684] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12084.278091] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12091.090175] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [12094.434180] LustreError: 423418:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12094.452314] LustreError: 423418:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [12095.462788] LustreError: 423419:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12100.600854] LustreError: 423418:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12102.273349] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12102.885485] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12109.762291] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [12119.038668] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12126.362422] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12133.972336] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12134.375992] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12141.744732] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [12150.521324] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12155.899540] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [12160.996401] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12161.005385] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12161.036425] Lustre: Skipped 1 previous similar message [12162.721680] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.28@tcp (stopping) [12162.730509] Lustre: Skipped 2 previous similar messages [12166.112337] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12166.126629] Lustre: Skipped 2 previous similar messages [12166.322439] LustreError: 425998:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGS: namespace resource [0x70616d65646f6e:0x4:0x0].0x0 (00000000630c5fa1) refcount nonzero (1) after lock cleanup; forcing cleanup. [12167.796352] LustreError: 423414:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12168.434669] Lustre: server umount lustre-MDT0000 complete [12171.446867] LustreError: 423401:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.28@tcp arrived at 1790007989 with bad export cookie 11308034378169827985 [12171.471253] LustreError: 423401:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12173.600157] LustreError: 423399:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790007991 with bad export cookie 11308034378169827320 [12173.621066] LustreError: 423399:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12173.627209] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12176.352732] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12176.355755] LustreError: 423418:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12176.369792] Lustre: Skipped 1 previous similar message [12176.370478] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12176.417127] LustreError: 423418:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [12179.946941] Lustre: server umount lustre-OST0000 complete [12187.949838] Lustre: server umount lustre-MDT0001 complete [12201.627103] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [12204.291668] Key type lgssc unregistered [12204.500467] LNet: 427104:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12204.507586] LNetError: 427104:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12204.548931] LNet: Removed LNI 192.168.204.128@tcp [12205.558390] Key type .llcrypt unregistered [12205.561167] Key type ._llcrypt unregistered [12234.661779] Key type ._llcrypt registered [12234.670394] Key type .llcrypt registered [12234.834053] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [12236.082763] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12236.171393] alg: No test for adler32 (adler32-zlib) [12237.248970] Lustre: Lustre: Build Version: 2.17.58_63_g3df9749 [12237.465525] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [12239.183405] Key type lgssc registered [12240.655343] Lustre: Echo OBD driver; http://www.lustre.org/ [12250.283599] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12250.309062] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12251.922510] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12258.090674] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [12261.352501] LustreError: 428538:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12261.387745] LustreError: 428538:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [12266.464711] LustreError: 428537:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12269.694454] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12270.127205] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12275.310148] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [12284.555287] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12292.450551] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12301.337273] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12301.804084] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12309.719457] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [12319.589896] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12321.270173] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [12330.649188] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.28@tcp (stopping) [12331.489216] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12331.514604] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12335.728070] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.28@tcp (stopping) [12335.733702] Lustre: Skipped 1 previous similar message [12340.852260] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.28@tcp (stopping) [12340.867797] Lustre: Skipped 2 previous similar messages [12342.239263] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12342.400632] Lustre: server umount lustre-MDT0001 complete [12346.866921] LustreError: 428533:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12346.904579] LustreError: 428533:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12351.969587] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12351.980036] LustreError: 428537:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12351.994247] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12351.994267] Lustre: Skipped 1 previous similar message [12351.997954] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12352.051511] Lustre: Skipped 2 previous similar messages [12355.598523] Lustre: server umount lustre-OST0000 complete [12357.092411] LustreError: 429266:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12360.687794] Lustre: server umount lustre-MDT0000 complete [12379.303585] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [12383.409462] Key type lgssc unregistered [12383.948858] LNet: 432224:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12383.954561] LNetError: 432224:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12383.981176] LNet: Removed LNI 192.168.204.128@tcp [12385.207347] Key type .llcrypt unregistered [12385.210121] Key type ._llcrypt unregistered [12416.879343] Key type ._llcrypt registered [12416.885411] Key type .llcrypt registered [12417.020906] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_hostid [12435.087736] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [12436.633659] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12436.710744] alg: No test for adler32 (adler32-zlib) [12437.804323] Lustre: Lustre: Build Version: 2.17.58_63_g3df9749 [12438.060918] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [12439.735334] Key type lgssc registered [12441.343817] Lustre: Echo OBD driver; http://www.lustre.org/ [12485.689034] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [12505.995283] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12506.025364] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12507.284688] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12507.317753] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12507.403712] Lustre: lustre-MDT0000: new disk, initializing [12507.585409] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12507.614945] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12513.535468] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [12532.463219] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12532.863797] Lustre: lustre-OST0000: new disk, initializing [12532.869558] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12532.874859] Lustre: Skipped 1 previous similar message [12532.879801] Lustre: 437275:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12533.002409] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12541.139787] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [12541.161364] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [12541.165247] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [12541.213200] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [12552.763465] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12570.401051] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12570.698555] Lustre: 436257:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12570.872789] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12570.986309] Lustre: lustre-MDT0001: new disk, initializing [12571.120342] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12571.171272] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [12571.188956] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [12577.368409] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [12589.329065] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [12589.643546] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [12595.357662] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [12595.614304] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [12602.863505] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12602.865232] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12602.866468] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12602.912832] Lustre: Skipped 1 previous similar message [12605.585876] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.28@tcp (stopping) [12605.593108] Lustre: Skipped 1 previous similar message [12607.985862] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12608.001391] Lustre: Skipped 1 previous similar message [12610.679511] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.28@tcp (stopping) [12615.139464] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12615.439817] Lustre: server umount lustre-MDT0001 complete [12623.343064] LustreError: 436269:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12623.366715] LustreError: 436269:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [12628.457617] LustreError: 436268:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12628.489721] LustreError: 436268:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12633.597805] LustreError: 436269:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12633.631957] LustreError: 436269:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12638.688609] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12638.692905] LustreError: 436268:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12638.701312] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12638.706325] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12638.749538] Lustre: Skipped 2 previous similar messages [12640.413596] Lustre: server umount lustre-OST0000 complete [12643.832116] LustreError: 436269:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12659.167151] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12659.491918] Lustre: server umount lustre-MDT0000 complete [12676.007311] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [12679.637944] Key type lgssc unregistered [12680.031894] LNet: 441030:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12680.039671] LNetError: 441030:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12680.063531] LNet: Removed LNI 192.168.204.128@tcp [12681.246623] Key type .llcrypt unregistered [12681.249831] Key type ._llcrypt unregistered [12730.323549] Key type ._llcrypt registered [12730.326919] Key type .llcrypt registered [12730.478379] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [12730.903504] Key type .llcrypt unregistered [12730.905211] Key type ._llcrypt unregistered [12749.757785] Key type ._llcrypt registered [12749.759418] Key type .llcrypt registered [12749.910134] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [12750.540520] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12750.571768] alg: No test for adler32 (adler32-zlib) [12751.617355] Lustre: Lustre: Build Version: 2.17.58_63_g3df9749 [12751.844642] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [12753.583435] Key type lgssc registered [12755.188365] Lustre: Echo OBD driver; http://www.lustre.org/ [12788.622044] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12788.668555] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12790.003931] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12790.049396] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12790.182095] Lustre: lustre-MDT0000: new disk, initializing [12790.346657] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12790.363307] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12796.573179] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [12809.973704] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12810.200990] Lustre: 444440:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12810.353312] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12810.374034] Lustre: Skipped 1 previous similar message [12810.523152] Lustre: lustre-MDT0001: new disk, initializing [12810.683427] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12810.758798] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12810.778121] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12816.567327] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [12828.521326] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12835.483722] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12843.339250] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12843.762881] Lustre: lustre-OST0000: new disk, initializing [12843.768800] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12843.788291] Lustre: 446410:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12843.926518] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12850.312740] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12850.328180] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12850.476650] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12851.518777] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [12862.561173] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12871.151819] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12871.153046] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12871.177915] Lustre: Skipped 2 previous similar messages [12871.201040] Lustre: Skipped 2 previous similar messages [12875.749908] Lustre: server umount lustre-MDT0000 complete [12880.077578] LustreError: 444431:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790008698 with bad export cookie 14262918745782307869 [12880.082871] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12880.099520] LustreError: 444431:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12880.774303] Lustre: server umount lustre-MDT0001 complete [12895.808373] Lustre: server umount lustre-OST0000 complete [12907.598299] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 12:38:42 (1790008722) [12914.480745] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12914.881869] LustreError: 448438:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12914.903459] LustreError: 448438:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [12915.004584] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12919.485629] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [12920.304158] LustreError: 448439:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12924.391486] LustreError: 448438:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12929.259755] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12929.513667] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [12929.653563] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12935.080046] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [12943.282281] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12950.012608] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12969.977038] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12970.216848] Lustre: Found index 0 for lustre-OST0000, updating log [12970.395584] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12977.580389] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [12985.787153] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [12985.995825] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12990.947921] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12990.977860] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12993.075467] Lustre: server umount lustre-OST0000 complete [13011.428263] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13011.441508] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13011.466376] Lustre: Skipped 1 previous similar message [13011.483857] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13011.500515] Lustre: Skipped 3 previous similar messages [13013.478397] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13015.731919] Lustre: server umount lustre-MDT0000 complete [13018.597826] LustreError: 448439:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13018.641375] LustreError: 448439:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13020.229725] LustreError: 449174:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790008838 with bad export cookie 14262918745782309262 [13020.235516] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13020.243925] LustreError: 449174:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13020.811784] Lustre: server umount lustre-MDT0001 complete [13031.077941] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 12:40:46 (1790008846) [13040.607977] LNet: 1 local NIs in recovery (showing 1): 192.168.204.128@tcp [13657.636629] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13658.047154] LustreError: 454852:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13658.062823] LustreError: 454852:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [13658.122919] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13663.102180] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [13663.202167] LustreError: 454853:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13668.328882] LustreError: 454852:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13674.459113] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13674.464912] LustreError: 454853:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13674.483851] LustreError: 454853:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13674.770698] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13679.879832] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [13688.711948] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13695.429466] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13716.213699] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13716.680117] Lustre: Found index 0 for lustre-OST0000, updating log [13717.111543] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13719.524190] LNet: 1 local NIs in recovery (showing 1): 192.168.204.128@tcp [13724.738871] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [13733.357612] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13736.547964] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13736.573889] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13736.582737] Lustre: Skipped 2 previous similar messages [13741.747747] Lustre: server umount lustre-OST0000 complete [13746.658328] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13746.677610] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13746.699876] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13747.745504] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13747.748110] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13749.922701] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13749.928299] Lustre: Skipped 3 previous similar messages [13752.225950] Lustre: server umount lustre-MDT0000 complete [13752.995425] LustreError: 454849:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13753.015706] LustreError: 454849:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13756.693904] LustreError: 454833:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790009574 with bad export cookie 14262918745782310347 [13756.713705] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13756.722436] LustreError: 454833:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13757.608825] Lustre: server umount lustre-MDT0001 complete [13779.935582] LNet: 1 local NIs in recovery (showing 1): 192.168.204.128@tcp [13791.202333] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_hostid [13804.656522] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [13848.743587] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 12:54:23 (1790009663) [13870.875888] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [13874.489924] Key type lgssc unregistered [13874.892169] LNet: 461470:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13874.904905] LNetError: 461470:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13977.320586] LNet: Removed LNI 192.168.204.128@tcp [13978.142060] Key type .llcrypt unregistered [13978.143646] Key type ._llcrypt unregistered [13997.529637] Key type ._llcrypt registered [13997.532512] Key type .llcrypt registered [13997.683693] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [13999.042635] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13999.159790] alg: No test for adler32 (adler32-zlib) [14000.480610] Lustre: Lustre: Build Version: 2.17.58_63_g3df9749 [14000.800243] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [14002.495461] Key type lgssc registered [14003.963318] Lustre: Echo OBD driver; http://www.lustre.org/ [14009.289079] LNet: Added LNI 192.168.204.128@tcp100 [8/256/0/180] [14010.226191] LNet: Added LNI 192.168.204.128@tcp101 [8/256/0/180] [14011.194559] LNet: Added LNI 192.168.204.128@tcp102 [8/256/0/180] [14012.091827] LNet: Added LNI 192.168.204.128@tcp103 [8/256/0/180] [14012.981907] LNet: Added LNI 192.168.204.128@tcp104 [8/256/0/180] [14013.845555] LNet: Added LNI 192.168.204.128@tcp105 [8/256/0/180] [14014.765057] LNet: Added LNI 192.168.204.128@tcp106 [8/256/0/180] [14015.698822] LNet: Added LNI 192.168.204.128@tcp107 [8/256/0/180] [14016.538445] LNet: Added LNI 192.168.204.128@tcp108 [8/256/0/180] [14017.374935] LNet: Added LNI 192.168.204.128@tcp109 [8/256/0/180] [14018.180444] LNet: Added LNI 192.168.204.128@tcp110 [8/256/0/180] [14019.160764] LNet: Added LNI 192.168.204.128@tcp111 [8/256/0/180] [14020.110871] LNet: Added LNI 192.168.204.128@tcp112 [8/256/0/180] [14021.242166] LNet: Added LNI 192.168.204.128@tcp113 [8/256/0/180] [14022.650128] LNet: Added LNI 192.168.204.128@tcp114 [8/256/0/180] [14023.753903] LNet: Added LNI 192.168.204.128@tcp115 [8/256/0/180] [14025.006347] LNet: Added LNI 192.168.204.128@tcp116 [8/256/0/180] [14026.174889] LNet: Added LNI 192.168.204.128@tcp117 [8/256/0/180] [14027.073821] LNet: Added LNI 192.168.204.128@tcp118 [8/256/0/180] [14027.901666] LNet: Added LNI 192.168.204.128@tcp119 [8/256/0/180] [14028.858169] LNet: Added LNI 192.168.204.128@tcp120 [8/256/0/180] [14030.012722] LNet: Added LNI 192.168.204.128@tcp121 [8/256/0/180] [14031.012716] LNet: Added LNI 192.168.204.128@tcp122 [8/256/0/180] [14032.003791] LNet: Added LNI 192.168.204.128@tcp123 [8/256/0/180] [14033.120809] LNet: Added LNI 192.168.204.128@tcp124 [8/256/0/180] [14034.231817] LNet: Added LNI 192.168.204.128@tcp125 [8/256/0/180] [14035.285338] LNet: Added LNI 192.168.204.128@tcp126 [8/256/0/180] [14036.778863] LNet: Added LNI 192.168.204.128@tcp127 [8/256/0/180] [14037.822676] LNet: Added LNI 192.168.204.128@tcp128 [8/256/0/180] [14038.987405] LNet: Added LNI 192.168.204.128@tcp129 [8/256/0/180] [14040.122187] LNet: Added LNI 192.168.204.128@tcp130 [8/256/0/180] [14040.917108] LNet: Added LNI 192.168.204.128@tcp131 [8/256/0/180] [14041.770563] LNet: Added LNI 192.168.204.128@tcp132 [8/256/0/180] [14042.841673] LNet: Added LNI 192.168.204.128@tcp133 [8/256/0/180] [14043.553855] LNet: Added LNI 192.168.204.128@tcp134 [8/256/0/180] [14044.503896] LNet: Added LNI 192.168.204.128@tcp135 [8/256/0/180] [14051.559562] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14051.599103] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14052.763581] Lustre: 464520:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.128@tcp: can't add NID 192.168.204.128@tcp130: rc = -75 [14052.922673] LNetError: 461966:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.128 [14052.946856] LNetError: Protocol error connecting to 192.168.204.128@tcp116 at host 192.168.204.128:988: is it running a compatible version of Lustre? [14052.958749] LNetError: Skipped 2 previous similar messages [14053.026462] LNet: 4 local NIs in recovery (showing 4): 192.168.204.128@tcp124, 192.168.204.128@tcp125, 192.168.204.128@tcp126, 192.168.204.128@tcp127 [14053.144848] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14053.191525] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14053.289377] Lustre: lustre-MDT0000: new disk, initializing [14053.413850] Lustre: 464576:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.128@tcp: can't add NID 192.168.204.128@tcp130: rc = -75 [14053.422895] Lustre: 464576:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [14053.465087] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14053.489065] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14055.090883] LNetError: 464541:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.128 [14055.108646] LNetError: 464541:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 15 previous similar messages [14055.115352] LNetError: Protocol error connecting to 192.168.204.128@tcp125 at host 192.168.204.128:988: is it running a compatible version of Lustre? [14055.129516] LNetError: Skipped 14 previous similar messages [14057.768441] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [14059.232972] LNetError: 461966:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.128 [14059.248327] LNetError: 461966:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 2 previous similar messages [14059.253033] LNetError: Protocol error connecting to 192.168.204.128@tcp125 at host 192.168.204.128:988: is it running a compatible version of Lustre? [14067.430518] LNetError: 461969:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.128 [14067.442125] LNetError: Protocol error connecting to 192.168.204.128@tcp125 at host 192.168.204.128:988: is it running a compatible version of Lustre? [14068.675688] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14068.867268] Lustre: 464538:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14068.918255] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14068.921609] Lustre: Skipped 1 previous similar message [14068.981537] Lustre: lustre-MDT0001: new disk, initializing [14069.012700] Lustre: 465337:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.128@tcp: can't add NID 192.168.204.128@tcp130: rc = -75 [14069.024910] Lustre: 465337:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [14069.146101] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14069.185452] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14069.203623] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14074.392744] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [14077.999261] Lustre: 465720:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.128@tcp: can't add NID 192.168.204.128@tcp130: rc = -75 [14078.020923] Lustre: 465720:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14083.808826] LNetError: 464547:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.128 [14083.817940] LNetError: Protocol error connecting to 192.168.204.128@tcp125 at host 192.168.204.128:988: is it running a compatible version of Lustre? [14085.806099] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14091.832869] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14092.000366] LNetError: 461966:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.128 [14092.016836] LNetError: 461966:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 5 previous similar messages [14092.026685] LNetError: Protocol error connecting to 192.168.204.128@tcp112 at host 192.168.204.128:988: is it running a compatible version of Lustre? [14092.047966] LNetError: Skipped 5 previous similar messages [14100.155983] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14100.513768] Lustre: lustre-OST0000: new disk, initializing [14100.519981] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14100.534903] Lustre: 466509:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14100.713716] Lustre: 466514:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.128@tcp: can't add NID 192.168.204.128@tcp130: rc = -75 [14100.721294] Lustre: 466514:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [14100.798975] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14108.164428] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [14108.386397] LNetError: 464552:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.128 [14108.392208] LNetError: 464552:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 11 previous similar messages [14108.397037] LNetError: Protocol error connecting to 192.168.204.128@tcp126 at host 192.168.204.128:988: is it running a compatible version of Lustre? [14108.401518] LNetError: Skipped 11 previous similar messages [14110.270405] Lustre: 466967:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.128@tcp: can't add NID 192.168.204.128@tcp130: rc = -75 [14110.288731] Lustre: 466967:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14110.335306] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14110.348487] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14110.479512] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14112.481166] LNet: 1 local NIs in recovery (showing 1): 192.168.204.128@tcp125 [14118.941602] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14131.171606] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14131.185687] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14131.188274] LustreError: Skipped 1 previous similar message [14131.215577] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14131.231627] Lustre: Skipped 1 previous similar message [14134.026647] Lustre: server umount lustre-OST0000 complete [14140.897349] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14140.903865] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14140.924713] Lustre: Skipped 1 previous similar message [14140.937744] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14141.160756] LNetError: 464541:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.128 [14141.187202] LNetError: 464541:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 16 previous similar messages [14141.200855] LNetError: Protocol error connecting to 192.168.204.128@tcp117 at host 192.168.204.128:988: is it running a compatible version of Lustre? [14141.219387] LNetError: Skipped 16 previous similar messages [14144.613548] Lustre: server umount lustre-MDT0000 complete [14148.907137] LustreError: 466508:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790009966 with bad export cookie 13601061776120554753 [14148.910183] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14148.930698] LustreError: 466508:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [14149.442077] Lustre: server umount lustre-MDT0001 complete [14163.823658] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [14167.637749] Key type lgssc unregistered [14168.271785] LNet: 468424:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14168.278568] LNetError: 468424:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14168.285387] LNet: Removed LNI 192.168.204.128@tcp [14169.311173] LNet: Removed LNI 192.168.204.128@tcp100 [14169.313237] LNetError: 468424:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14169.329611] LNetError: 468424:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14169.360372] LNet: Removed LNI 192.168.204.128@tcp101 [14170.399652] LNet: Removed LNI 192.168.204.128@tcp102 [14170.421976] LNetError: 468424:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14170.429681] LNetError: 468424:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14170.461932] LNet: Removed LNI 192.168.204.128@tcp103 [14171.489363] LNet: Removed LNI 192.168.204.128@tcp104 [14172.511365] LNet: Removed LNI 192.168.204.128@tcp105 [14172.532867] LNetError: 468424:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14172.545281] LNetError: 468424:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [14172.552788] LNet: Removed LNI 192.168.204.128@tcp106 [14173.600439] LNet: Removed LNI 192.168.204.128@tcp107 [14174.623798] LNet: Removed LNI 192.168.204.128@tcp108 [14175.647348] LNet: Removed LNI 192.168.204.128@tcp109 [14176.671966] LNet: Removed LNI 192.168.204.128@tcp110 [14176.676939] LNetError: 468424:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14176.695184] LNetError: 468424:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [14176.711701] LNet: Removed LNI 192.168.204.128@tcp111 [14177.759323] LNet: Removed LNI 192.168.204.128@tcp112 [14178.783210] LNet: Removed LNI 192.168.204.128@tcp113 [14179.807918] LNet: Removed LNI 192.168.204.128@tcp114 [14180.831379] LNet: Removed LNI 192.168.204.128@tcp115 [14181.855233] LNet: Removed LNI 192.168.204.128@tcp116 [14182.879332] LNet: Removed LNI 192.168.204.128@tcp117 [14183.903192] LNet: Removed LNI 192.168.204.128@tcp118 [14184.927486] LNet: Removed LNI 192.168.204.128@tcp119 [14184.930801] LNetError: 468424:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14184.939471] LNetError: 468424:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [14184.962148] LNet: Removed LNI 192.168.204.128@tcp120 [14186.017072] LNet: Removed LNI 192.168.204.128@tcp121 [14187.039176] LNet: Removed LNI 192.168.204.128@tcp122 [14188.063245] LNet: Removed LNI 192.168.204.128@tcp123 [14189.087361] LNet: Removed LNI 192.168.204.128@tcp124 [14189.098352] LNet: Removed LNI 192.168.204.128@tcp125 [14190.111324] LNet: Removed LNI 192.168.204.128@tcp126 [14191.136080] LNet: Removed LNI 192.168.204.128@tcp127 [14192.159705] LNet: Removed LNI 192.168.204.128@tcp128 [14193.183888] LNet: Removed LNI 192.168.204.128@tcp129 [14193.186269] LNet: Removed LNI 192.168.204.128@tcp130 [14193.190688] LNet: Removed LNI 192.168.204.128@tcp131 [14193.194307] LNet: Removed LNI 192.168.204.128@tcp132 [14193.197415] LNet: Removed LNI 192.168.204.128@tcp133 [14193.199601] LNet: Removed LNI 192.168.204.128@tcp134 [14193.225033] LNet: Removed LNI 192.168.204.128@tcp135 [14194.364885] Key type .llcrypt unregistered [14194.373433] Key type ._llcrypt unregistered [14222.182746] Key type ._llcrypt registered [14222.185516] Key type .llcrypt registered [14222.350791] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_hostid [14238.321663] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [14239.180444] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14239.198051] alg: No test for adler32 (adler32-zlib) [14240.403976] Lustre: Lustre: Build Version: 2.17.58_63_g3df9749 [14240.710979] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [14242.535296] Key type lgssc registered [14245.008382] Lustre: Echo OBD driver; http://www.lustre.org/ [14285.795193] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 13:01:40 (1790010100) [14306.792200] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [14310.742806] Key type lgssc unregistered [14311.109930] LNet: 472545:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14311.114892] LNetError: 472545:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14311.151931] LNet: Removed LNI 192.168.204.128@tcp [14312.526378] Key type .llcrypt unregistered [14312.534267] Key type ._llcrypt unregistered [14329.431391] Key type ._llcrypt registered [14329.437299] Key type .llcrypt registered [14329.561752] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [14332.022483] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14332.074645] alg: No test for adler32 (adler32-zlib) [14333.339819] Lustre: Lustre: Build Version: 2.17.58_63_g3df9749 [14333.744861] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [14335.519431] Key type lgssc registered [14336.995691] Lustre: Echo OBD driver; http://www.lustre.org/ [14350.619398] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14350.646538] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14352.011466] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14352.042762] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14352.134199] Lustre: lustre-MDT0000: new disk, initializing [14352.215198] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14352.254397] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14357.726836] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [14369.212874] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14369.370625] Lustre: 473775:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14369.444477] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14369.448280] Lustre: Skipped 1 previous similar message [14369.545160] Lustre: lustre-MDT0001: new disk, initializing [14369.669879] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14369.720106] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14369.735141] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14374.281331] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [14383.559673] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14389.773884] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14396.465493] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14396.748964] Lustre: lustre-OST0000: new disk, initializing [14396.753495] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14396.762705] Lustre: 475742:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14396.911254] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14403.212454] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14403.228907] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14403.391288] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14404.399472] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [14415.683424] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14420.355911] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing lnet_if_list [14426.677693] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [14426.717097] LNet: Added LNI 192.168.204.128@tcp10 [8/256/0/180] [14432.894210] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [14432.928664] LNet: Added LNI 192.168.204.128@tcp11 [8/256/0/180] [14437.284641] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [14437.311937] LNet: Added LNI 192.168.204.128@tcp12 [8/256/0/180] [14441.809713] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [14441.848643] LNet: Added LNI 192.168.204.128@tcp13 [8/256/0/180] [14447.634901] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [14447.676900] LNet: Added LNI 192.168.204.128@tcp14 [8/256/0/180] [14453.318455] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [14453.347883] LNet: Added LNI 192.168.204.128@tcp15 [8/256/0/180] [14460.321596] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [14460.353553] LNet: Added LNI 192.168.204.128@tcp16 [8/256/0/180] [14466.210724] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [14466.288379] LNet: Added LNI 192.168.204.128@tcp17 [8/256/0/180] [14472.823478] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [14472.873443] LNet: Added LNI 192.168.204.128@tcp18 [8/256/0/180] [14480.316141] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [14480.361975] LNet: Added LNI 192.168.204.128@tcp19 [8/256/0/180] [14486.055671] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [14486.072417] LNet: Added LNI 192.168.204.128@tcp20 [8/256/0/180] [14491.211848] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [14491.268674] LNet: Added LNI 192.168.204.128@tcp21 [8/256/0/180] [14498.912739] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [14498.977290] LNet: Added LNI 192.168.204.128@tcp22 [8/256/0/180] [14506.603791] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [14506.651174] LNet: Added LNI 192.168.204.128@tcp23 [8/256/0/180] [14515.861433] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [14515.926233] LNet: Added LNI 192.168.204.128@tcp24 [8/256/0/180] [14522.299805] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [14522.346573] LNet: Added LNI 192.168.204.128@tcp25 [8/256/0/180] [14529.748503] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [14529.805062] LNet: Added LNI 192.168.204.128@tcp26 [8/256/0/180] [14536.702249] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [14536.732785] LNet: Added LNI 192.168.204.128@tcp27 [8/256/0/180] [14545.137462] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [14545.208486] LNet: Added LNI 192.168.204.128@tcp28 [8/256/0/180] [14553.159949] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [14553.206252] LNet: Added LNI 192.168.204.128@tcp29 [8/256/0/180] [14561.278981] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [14561.337939] LNet: Added LNI 192.168.204.128@tcp30 [8/256/0/180] [14569.635754] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [14569.692407] LNet: Added LNI 192.168.204.128@tcp31 [8/256/0/180] [14577.583275] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [14577.620874] LNet: Added LNI 192.168.204.128@tcp32 [8/256/0/180] [14585.821966] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [14585.860383] LNet: Added LNI 192.168.204.128@tcp33 [8/256/0/180] [14593.898070] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [14593.974270] LNet: Added LNI 192.168.204.128@tcp34 [8/256/0/180] [14602.900846] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [14602.930710] LNet: Added LNI 192.168.204.128@tcp35 [8/256/0/180] [14611.966028] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [14612.005888] LNet: Added LNI 192.168.204.128@tcp36 [8/256/0/180] [14621.881846] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [14621.920676] LNet: Added LNI 192.168.204.128@tcp37 [8/256/0/180] [14631.551388] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [14631.582457] LNet: Added LNI 192.168.204.128@tcp38 [8/256/0/180] [14641.182192] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [14641.214929] LNet: Added LNI 192.168.204.128@tcp39 [8/256/0/180] [14650.904260] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [14650.934539] LNet: Added LNI 192.168.204.128@tcp40 [8/256/0/180] [14661.132714] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [14661.193214] LNet: Added LNI 192.168.204.128@tcp41 [8/256/0/180] [14671.689730] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [14671.759422] LNet: Added LNI 192.168.204.128@tcp42 [8/256/0/180] [14681.697190] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [14681.718766] LNet: Added LNI 192.168.204.128@tcp43 [8/256/0/180] [14691.839416] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [14691.899558] LNet: Added LNI 192.168.204.128@tcp44 [8/256/0/180] [14701.643503] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [14701.708698] LNet: Added LNI 192.168.204.128@tcp45 [8/256/0/180] [14711.358763] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [14711.393318] LNet: Added LNI 192.168.204.128@tcp46 [8/256/0/180] [14721.818912] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [14721.861824] LNet: Added LNI 192.168.204.128@tcp47 [8/256/0/180] [14733.212969] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [14733.254531] LNet: Added LNI 192.168.204.128@tcp48 [8/256/0/180] [14745.004585] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [14745.034986] LNet: Added LNI 192.168.204.128@tcp49 [8/256/0/180] [14756.791388] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [14756.848972] LNet: Added LNI 192.168.204.128@tcp50 [8/256/0/180] [14779.360838] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14779.374394] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14779.402128] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14782.433609] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14782.442835] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14782.460157] Lustre: Skipped 1 previous similar message [14782.469635] Lustre: Skipped 2 previous similar messages [14787.553778] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14787.560736] Lustre: Skipped 1 previous similar message [14792.159720] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14792.586275] Lustre: server umount lustre-MDT0000 complete [14797.050980] LustreError: 473766:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790010615 with bad export cookie 2140188276612290139 [14797.078277] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14797.344353] Lustre: server umount lustre-MDT0001 complete [14813.155063] Lustre: 473036:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790010615/real 1790010615] req@00000000e5fd5fe7 x1876961684292992/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790010631 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14813.216898] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14818.336283] Lustre: 473036:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790010620/real 1790010620] req@00000000c2517e35 x1876961684293120/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790010636 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14820.983570] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [14824.419848] Lustre: 473038:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790010626/real 1790010626] req@000000004d4e50f1 x1876961684293248/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790010642 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14829.538431] Lustre: 473039:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790010631/real 1790010631] req@000000002f50152c x1876961684293504/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790010647 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14832.456339] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14844.210146] LustreError: 475735:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14844.243521] LustreError: 475735:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [14844.299818] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14847.397462] LustreError: 475736:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14847.399637] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14849.606555] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [14852.586374] LustreError: 475735:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14857.697162] LustreError: 475736:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14858.512768] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14858.845444] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14863.574786] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [14863.854370] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14872.919920] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14878.967260] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14900.711804] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14900.720766] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14900.748698] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14900.764464] Lustre: Skipped 1 previous similar message [14904.804344] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14904.806618] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14904.810619] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14904.868085] Lustre: Skipped 1 previous similar message [14906.992459] Lustre: server umount lustre-OST0000 complete [14915.040468] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14915.043064] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14915.076976] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14915.084926] Lustre: Skipped 1 previous similar message [14919.355632] Lustre: server umount lustre-MDT0000 complete [14920.170586] LustreError: 499998:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14920.198305] LustreError: 499998:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [14924.583217] LustreError: 473768:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790010742 with bad export cookie 2140188276612292876 [14924.604263] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14924.605467] LustreError: 473768:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [14925.429509] Lustre: server umount lustre-MDT0001 complete [14937.220204] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [14939.907471] Key type lgssc unregistered [14940.431988] LNet: 502800:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14940.439397] LNetError: 502800:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14940.446836] LNet: Removed LNI 192.168.204.128@tcp [14940.449218] LNet: Removed LNI 192.168.204.128@tcp10 [14940.451798] LNet: Removed LNI 192.168.204.128@tcp11 [14940.454068] LNet: Removed LNI 192.168.204.128@tcp12 [14940.456471] LNet: Removed LNI 192.168.204.128@tcp13 [14940.458581] LNet: Removed LNI 192.168.204.128@tcp14 [14940.460621] LNet: Removed LNI 192.168.204.128@tcp15 [14940.462818] LNet: Removed LNI 192.168.204.128@tcp16 [14940.464892] LNet: Removed LNI 192.168.204.128@tcp17 [14940.466888] LNet: Removed LNI 192.168.204.128@tcp18 [14940.468898] LNet: Removed LNI 192.168.204.128@tcp19 [14940.470900] LNet: Removed LNI 192.168.204.128@tcp20 [14940.473237] LNet: Removed LNI 192.168.204.128@tcp21 [14940.475115] LNet: Removed LNI 192.168.204.128@tcp22 [14940.477476] LNet: Removed LNI 192.168.204.128@tcp23 [14940.479836] LNet: Removed LNI 192.168.204.128@tcp24 [14940.482099] LNet: Removed LNI 192.168.204.128@tcp25 [14940.484408] LNet: Removed LNI 192.168.204.128@tcp26 [14940.486433] LNet: Removed LNI 192.168.204.128@tcp27 [14940.488826] LNet: Removed LNI 192.168.204.128@tcp28 [14940.491324] LNet: Removed LNI 192.168.204.128@tcp29 [14940.493391] LNet: Removed LNI 192.168.204.128@tcp30 [14940.495660] LNet: Removed LNI 192.168.204.128@tcp31 [14940.500174] LNet: Removed LNI 192.168.204.128@tcp32 [14940.505416] LNet: Removed LNI 192.168.204.128@tcp33 [14940.508544] LNet: Removed LNI 192.168.204.128@tcp34 [14940.512434] LNet: Removed LNI 192.168.204.128@tcp35 [14940.518929] LNet: Removed LNI 192.168.204.128@tcp36 [14940.523223] LNet: Removed LNI 192.168.204.128@tcp37 [14940.526678] LNet: Removed LNI 192.168.204.128@tcp38 [14940.529930] LNet: Removed LNI 192.168.204.128@tcp39 [14940.536283] LNet: Removed LNI 192.168.204.128@tcp40 [14940.539858] LNet: Removed LNI 192.168.204.128@tcp41 [14940.543267] LNet: Removed LNI 192.168.204.128@tcp42 [14940.545163] LNet: Removed LNI 192.168.204.128@tcp43 [14940.548978] LNet: Removed LNI 192.168.204.128@tcp44 [14940.552223] LNet: Removed LNI 192.168.204.128@tcp45 [14940.556570] LNet: Removed LNI 192.168.204.128@tcp46 [14940.560386] LNet: Removed LNI 192.168.204.128@tcp47 [14940.566368] LNet: Removed LNI 192.168.204.128@tcp48 [14940.573284] LNet: Removed LNI 192.168.204.128@tcp49 [14940.592175] LNet: Removed LNI 192.168.204.128@tcp50 [14941.556450] Key type .llcrypt unregistered [14941.559463] Key type ._llcrypt unregistered [14964.527853] Key type ._llcrypt registered [14964.529845] Key type .llcrypt registered [14964.606101] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_hostid [14982.949993] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [14984.905556] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14984.946723] alg: No test for adler32 (adler32-zlib) [14986.206313] Lustre: Lustre: Build Version: 2.17.58_63_g3df9749 [14986.523389] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [14988.272711] Key type lgssc registered [14990.483424] Lustre: Echo OBD driver; http://www.lustre.org/ [15030.775972] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15030.801867] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15032.085645] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15032.140872] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15032.284954] Lustre: lustre-MDT0000: new disk, initializing [15032.430466] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15032.459320] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15037.282963] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [15048.589324] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15048.740835] Lustre: 506014:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15048.827631] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15048.833471] Lustre: Skipped 1 previous similar message [15048.900191] Lustre: lustre-MDT0001: new disk, initializing [15049.005763] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15049.038350] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15049.056538] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15053.082598] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [15062.869919] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15069.628080] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15079.295086] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15079.618904] Lustre: *** cfs_fail_loc=90f, val=0*** [15079.754884] Lustre: lustre-OST0000: new disk, initializing [15079.758830] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15079.764217] Lustre: 508031:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15079.883336] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15087.446861] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [15087.757221] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15087.787265] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15087.971565] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15097.770765] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15108.588694] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15108.592182] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15108.598336] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15108.610956] Lustre: Skipped 1 previous similar message [15108.639969] Lustre: Skipped 1 previous similar message [15113.710983] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15118.822265] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15118.836203] Lustre: Skipped 2 previous similar messages [15120.351163] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15120.531728] Lustre: server umount lustre-OST0000 complete [15125.987368] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15126.015872] LustreError: Skipped 1 previous similar message [15126.021477] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15126.050292] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15129.057683] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15134.182621] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15134.191275] Lustre: Skipped 3 previous similar messages [15139.295174] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15139.531631] Lustre: server umount lustre-MDT0000 complete [15143.453450] LustreError: 506007:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790010961 with bad export cookie 9595298049011508268 [15143.455890] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15143.459354] LustreError: 506007:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [15143.697223] Lustre: server umount lustre-MDT0001 complete [15155.285416] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [15158.568719] Key type lgssc unregistered [15158.940524] LNet: 509940:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15158.955877] LNetError: 509940:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15158.981274] LNet: Removed LNI 192.168.204.128@tcp [15159.919328] Key type .llcrypt unregistered [15159.922296] Key type ._llcrypt unregistered [15181.835227] Key type ._llcrypt registered [15181.838267] Key type .llcrypt registered [15181.960800] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [15182.718224] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15182.775634] alg: No test for adler32 (adler32-zlib) [15183.779257] Lustre: Lustre: Build Version: 2.17.58_63_g3df9749 [15184.154953] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [15185.880082] Key type lgssc registered [15187.163366] Lustre: Echo OBD driver; http://www.lustre.org/ [15196.409818] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15196.431752] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15197.975523] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15203.398523] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [15207.397985] LustreError: 511225:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15207.432664] LustreError: 511225:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [15212.520168] LustreError: 511226:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15213.096121] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15213.459441] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15218.689268] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [15226.807637] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15233.202294] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15241.352636] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15241.978697] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15250.206297] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [15258.802309] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15270.930505] loop0: detected capacity change from 0 to 204800000 [15271.282108] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15271.294961] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15271.310653] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15278.766779] loop0: detected capacity change from 0 to 204800000 [15279.113376] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15279.122659] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15279.133652] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15285.544139] loop0: detected capacity change from 0 to 204800000 [15292.202336] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15292.337341] Lustre: fs15246-OST0000: new disk, initializing [15292.345450] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [15292.351899] Lustre: 514825:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15292.463086] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15299.955512] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [15310.876049] loop1: detected capacity change from 0 to 204800000 [15317.881079] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15318.050046] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [15318.160704] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [15318.289148] Lustre: fs15246-MDT0000: new disk, initializing [15318.508621] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15318.552181] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15318.726262] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [15319.791523] Lustre: 515948:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000194fd1a5 x1876962576484480/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1790011153 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [15324.661471] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [15324.897965] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [15324.909378] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [15324.961601] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [15350.262316] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15350.290491] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [15355.322708] Lustre: server umount fs15246-MDT0000 complete [15355.377689] LustreError: 513104:0:(ldlm_lib.c:1199:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15360.507380] LustreError: 511221:0:(ldlm_lib.c:1199:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15365.606223] LustreError: 511221:0:(ldlm_lib.c:1199:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15366.304647] Lustre: server umount fs15246-OST0000 complete [15380.962516] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15380.979905] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15386.112211] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15386.119060] Lustre: Skipped 2 previous similar messages [15387.080337] Lustre: server umount lustre-OST0000 complete [15392.736690] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15392.750628] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15392.786022] Lustre: Skipped 1 previous similar message [15392.794266] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15396.337324] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15398.399313] Lustre: server umount lustre-MDT0000 complete [15401.441041] LustreError: 511220:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15401.459961] LustreError: 511220:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [15402.726169] LustreError: 511208:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790011220 with bad export cookie 12090064293120927042 [15402.728070] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15402.742065] LustreError: 511208:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15403.152421] Lustre: server umount lustre-MDT0001 complete [15416.460939] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [15419.108708] Key type lgssc unregistered [15419.402425] LNet: 519044:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15419.415918] LNetError: 519044:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15419.442661] LNet: Removed LNI 192.168.204.128@tcp [15420.364356] Key type .llcrypt unregistered [15420.367887] Key type ._llcrypt unregistered [15441.503955] Key type ._llcrypt registered [15441.509362] Key type .llcrypt registered [15441.673826] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [15442.985711] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15443.089881] alg: No test for adler32 (adler32-zlib) [15444.362671] Lustre: Lustre: Build Version: 2.17.58_63_g3df9749 [15444.798264] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [15446.519215] Key type lgssc registered [15447.737376] Lustre: Echo OBD driver; http://www.lustre.org/ [15488.990944] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_hostid [15499.068754] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [15545.587307] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15545.603107] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15546.885470] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15546.929298] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15546.999759] Lustre: lustre-MDT0000: new disk, initializing [15547.133304] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15547.159993] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15551.795437] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [15563.786152] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15563.935679] Lustre: 523525:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15563.995334] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15564.002091] Lustre: Skipped 1 previous similar message [15564.055857] Lustre: lustre-MDT0001: new disk, initializing [15564.146776] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15564.173325] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15564.190725] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15567.916271] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [15578.379851] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15585.860195] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15595.458620] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15595.868600] Lustre: lustre-OST0000: new disk, initializing [15595.873709] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15595.890383] Lustre: 525498:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15596.012923] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15602.405515] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15602.428462] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15602.563325] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15602.973221] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [15613.805467] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15621.362485] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [15621.751463] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [15627.474860] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [15627.707582] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [15633.379544] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15633.380046] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15633.406471] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15633.424374] Lustre: Skipped 1 previous similar message [15636.320312] Lustre: server umount lustre-OST0000 complete [15641.059259] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15641.083978] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15641.098783] Lustre: Skipped 1 previous similar message [15641.106955] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15643.616062] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15643.620416] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15648.736775] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15648.750405] Lustre: Skipped 2 previous similar messages [15653.858454] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15653.864048] Lustre: Skipped 1 previous similar message [15654.368649] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15654.510178] Lustre: server umount lustre-MDT0000 complete [15660.655119] LustreError: 523519:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790011478 with bad export cookie 17518839126465111563 [15660.680397] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15660.686148] LustreError: 523519:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15661.681727] Lustre: server umount lustre-MDT0001 complete [15691.844945] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_hostid [15705.691905] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [15745.711415] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 15613 sec ========= 13:26:01 (1790011561) [15747.674351] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 13:26:02 (1790011562) === [15751.796447] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 13:26:06 (1790011566) === [15804.554829] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [15808.286852] Key type lgssc unregistered [15808.695908] LNet: 532148:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15808.704125] LNetError: 532148:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15808.726604] LNet: Removed LNI 192.168.204.128@tcp [15809.832412] Key type .llcrypt unregistered [15809.840256] Key type ._llcrypt unregistered