[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 474668236 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.003277] x2apic enabled [ 0.004022] Switched APIC routing to physical x2apic. [ 0.005017] kvm-guest: setup PV IPIs [ 0.007759] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008022] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009013] pid_max: default: 32768 minimum: 301 [ 0.010135] LSM: Security Framework initializing [ 0.011063] Yama: becoming mindful. [ 0.012055] SELinux: Initializing. [ 0.013071] *** VALIDATE selinux *** [ 0.021301] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026168] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027152] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028103] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029109] *** VALIDATE tmpfs *** [ 0.030463] *** VALIDATE proc *** [ 0.032229] *** VALIDATE cgroup *** [ 0.033011] *** VALIDATE cgroup2 *** [ 0.035243] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036164] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038031] Spectre V2 : User space: Vulnerable [ 0.039011] Speculative Store Bypass: Vulnerable [ 0.042302] debug: unmapping init [mem 0xffffffffa6259000-0xffffffffa6260fff] [ 0.044181] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045703] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046025] ... version: 2 [ 0.047016] ... bit width: 48 [ 0.048018] ... generic registers: 4 [ 0.049018] ... value mask: 0000ffffffffffff [ 0.050018] ... max period: 00007fffffffffff [ 0.051016] ... fixed-purpose events: 3 [ 0.052014] ... event mask: 000000070000000f [ 0.053405] rcu: Hierarchical SRCU implementation. [ 0.055691] smp: Bringing up secondary CPUs ... [ 0.056639] x86: Booting SMP configuration: [ 0.057037] .... node #0, CPUs: #1 #2 #3 [ 0.061063] smp: Brought up 1 node, 4 CPUs [ 0.063017] smpboot: Max logical packages: 1 [ 0.064019] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.275498] node 0 deferred pages initialised in 209ms [ 0.279012] devtmpfs: initialized [ 0.280238] x86/mm: Memory block size: 128MB [ 0.282957] gcov: version magic: 0x41383552 [ 0.284333] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.288083] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.291373] pinctrl core: initialized pinctrl subsystem [ 0.293156] [ 0.293723] ************************************************************* [ 0.296013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.298073] ** ** [ 0.301011] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.303012] ** ** [ 0.305014] ** This means that this kernel is built to expose internal ** [ 0.307014] ** IOMMU data structures, which may compromise security on ** [ 0.310013] ** your system. ** [ 0.312012] ** ** [ 0.314014] ** If you see this message and you are not debugging the ** [ 0.317015] ** kernel, report this immediately to your vendor! ** [ 0.319013] ** ** [ 0.322013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.324013] ************************************************************* [ 0.327646] NET: Registered protocol family 16 [ 0.329518] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.332061] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.335065] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.339083] cpuidle: using governor menu [ 0.341259] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.344669] PCI: Using configuration type 1 for base access [ 0.347010] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.358084] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.359025] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.361094] cryptd: max_cpu_qlen set to 1000 [ 0.365077] ACPI: Added _OSI(Module Device) [ 0.366020] ACPI: Added _OSI(Processor Device) [ 0.368019] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.370016] ACPI: Added _OSI(Processor Aggregator Device) [ 0.376639] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.383375] ACPI: Interpreter enabled [ 0.384060] ACPI: PM: (supports S0 S3 S4 S5) [ 0.386015] ACPI: Using IOAPIC for interrupt routing [ 0.387139] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.391410] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.401794] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.403042] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.406018] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.408085] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.414420] acpiphp: Slot [2] registered [ 0.415116] acpiphp: Slot [5] registered [ 0.417149] acpiphp: Slot [6] registered [ 0.419128] acpiphp: Slot [7] registered [ 0.420192] acpiphp: Slot [8] registered [ 0.422124] acpiphp: Slot [9] registered [ 0.423117] acpiphp: Slot [10] registered [ 0.425131] acpiphp: Slot [3] registered [ 0.426132] acpiphp: Slot [4] registered [ 0.428127] acpiphp: Slot [11] registered [ 0.430119] acpiphp: Slot [12] registered [ 0.432119] acpiphp: Slot [13] registered [ 0.434123] acpiphp: Slot [14] registered [ 0.436118] acpiphp: Slot [15] registered [ 0.437115] acpiphp: Slot [16] registered [ 0.439284] acpiphp: Slot [17] registered [ 0.441117] acpiphp: Slot [18] registered [ 0.443121] acpiphp: Slot [19] registered [ 0.445128] acpiphp: Slot [20] registered [ 0.447180] acpiphp: Slot [21] registered [ 0.449119] acpiphp: Slot [22] registered [ 0.450118] acpiphp: Slot [23] registered [ 0.452114] acpiphp: Slot [24] registered [ 0.454118] acpiphp: Slot [25] registered [ 0.456176] acpiphp: Slot [26] registered [ 0.457142] acpiphp: Slot [27] registered [ 0.459179] acpiphp: Slot [28] registered [ 0.461129] acpiphp: Slot [29] registered [ 0.463119] acpiphp: Slot [30] registered [ 0.465148] acpiphp: Slot [31] registered [ 0.466068] PCI host bridge to bus 0000:00 [ 0.468016] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.470024] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.473026] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.476027] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.478026] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.481034] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.483257] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.487181] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.490371] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.501017] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.506061] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.509019] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.511080] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.514017] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.517462] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.519648] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.522035] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.525880] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.534017] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.549020] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.556015] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.563270] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.570017] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.576018] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.591017] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.609029] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.621017] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.631019] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.659017] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.673048] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.682016] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.689013] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.704016] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.714307] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.721015] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.727014] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.744026] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.761028] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.772015] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.781017] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.808016] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.819232] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.827015] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.835015] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.851017] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.863527] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.866397] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.868255] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.871434] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.873199] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.878032] iommu: Default domain type: Passthrough [ 0.879410] SCSI subsystem initialized [ 0.881287] ACPI: bus type USB registered [ 0.883129] usbcore: registered new interface driver usbfs [ 0.885110] usbcore: registered new interface driver hub [ 0.887092] usbcore: registered new device driver usb [ 0.889152] pps_core: LinuxPPS API ver. 1 registered [ 0.891010] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.894063] PTP clock support registered [ 0.896127] EDAC MC: Ver: 3.0.0 [ 0.898548] PCI: Using ACPI for IRQ routing [ 0.900711] NetLabel: Initializing [ 0.903012] NetLabel: domain hash size = 128 [ 0.904010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.906089] NetLabel: unlabeled traffic allowed by default [ 0.909137] vgaarb: loaded [ 0.910223] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.912014] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.920738] clocksource: Switched to clocksource kvm-clock [ 1.021883] VFS: Disk quotas dquot_6.6.0 [ 1.024421] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.027921] *** VALIDATE ramfs *** [ 1.029548] *** VALIDATE hugetlbfs *** [ 1.032446] pnp: PnP ACPI init [ 1.035342] pnp: PnP ACPI: found 6 devices [ 1.055083] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.059519] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.061948] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.064449] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.066776] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.069529] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.072658] NET: Registered protocol family 2 [ 1.074747] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.079700] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.083706] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.089892] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.093735] TCP: Hash tables configured (established 65536 bind 65536) [ 1.097865] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.100969] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.103787] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.107178] NET: Registered protocol family 1 [ 1.109930] RPC: Registered named UNIX socket transport module. [ 1.113280] RPC: Registered udp transport module. [ 1.116206] RPC: Registered tcp transport module. [ 1.118803] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.122441] NET: Registered protocol family 44 [ 1.124298] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.126568] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.128989] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.131733] PCI: CLS 0 bytes, default 64 [ 1.133556] Unpacking initramfs... [ 2.518158] debug: unmapping init [mem 0xffff983fbcc54000-0xffff983fbffbffff] [ 2.522235] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.524603] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.528837] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.020778] Initialise system trusted keyrings [ 3.022680] Key type blacklist registered [ 3.024691] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.038831] zbud: loaded [ 3.043153] *** VALIDATE nfs *** [ 3.044392] *** VALIDATE nfs4 *** [ 3.045850] pstore: using deflate compression [ 3.049754] Platform Keyring initialized [ 3.145270] NET: Registered protocol family 38 [ 3.146541] Key type asymmetric registered [ 3.147856] Asymmetric key parser 'x509' registered [ 3.149812] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.154110] io scheduler mq-deadline registered [ 3.155812] io scheduler kyber registered [ 3.157829] io scheduler bfq registered [ 3.160086] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.163847] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.166942] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.170106] ACPI: Power Button [PWRF] [ 3.175992] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.182367] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.196891] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.204459] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.215539] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.245709] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.273345] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.277383] Non-volatile memory driver v1.3 [ 3.279307] Linux agpgart interface v0.103 [ 3.308589] virtio_blk virtio1: [vda] 139528 512-byte logical blocks (71.4 MB/68.1 MiB) [ 3.312734] vda: detected capacity change from 0 to 71438336 [ 3.326405] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.329526] vdb: detected capacity change from 0 to 1073741824 [ 3.342947] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.345462] vdc: detected capacity change from 0 to 2621440000 [ 3.358654] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.361076] vdd: detected capacity change from 0 to 2621440000 [ 3.376825] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.379250] vde: detected capacity change from 0 to 4294967296 [ 3.391703] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.394302] vdf: detected capacity change from 0 to 4294967296 [ 3.399473] libphy: Fixed MDIO Bus: probed [ 3.404404] usbcore: registered new interface driver usbserial_generic [ 3.406447] usbserial: USB Serial support registered for generic [ 3.408309] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.412899] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.414647] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.416869] mousedev: PS/2 mouse device common for all mice [ 3.420746] rtc_cmos 00:05: RTC can wake from S4 [ 3.423613] rtc_cmos 00:05: registered as rtc0 [ 3.425309] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.426535] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.428566] intel_pstate: CPU model not supported [ 3.435614] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.437331] hid: raw HID events driver (C) Jiri Kosina [ 3.441886] usbcore: registered new interface driver usbhid [ 3.444209] usbhid: USB HID core driver [ 3.446123] drop_monitor: Initializing network drop monitor service [ 3.446140] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.447669] Initializing XFRM netlink socket [ 3.450641] NET: Registered protocol family 10 [ 3.452754] Segment Routing with IPv6 [ 3.453659] NET: Registered protocol family 17 [ 3.454959] mpls_gso: MPLS GSO support [ 3.459473] RAS: Correctable Errors collector initialized. [ 3.462616] AVX version of gcm_enc/dec engaged. [ 3.464701] AES CTR mode by8 optimization enabled [ 3.532848] sched_clock: Marking stable (3532820986, 0)->(4461531110, -928710124) [ 3.536709] registered taskstats version 1 [ 3.539682] Loading compiled-in X.509 certificates [ 3.542609] zswap: loaded using pool lzo/zbud [ 3.566434] Key type big_key registered [ 3.578055] Key type encrypted registered [ 3.579816] ima: No TPM chip found, activating TPM-bypass! [ 3.582032] ima: Allocated hash algorithm: sha1 [ 3.583930] ima: No architecture policies found [ 3.585809] evm: Initialising EVM extended attributes: [ 3.587974] evm: security.selinux [ 3.588951] evm: security.ima [ 3.590213] evm: security.capability [ 3.591677] evm: HMAC attrs: 0x1 [ 3.594170] rtc_cmos 00:05: setting system clock to 2026-06-09 09:20:38 UTC (1780996838) [ 3.600290] debug: unmapping init [mem 0xffffffffa7203000-0xffffffffa73fffff] [ 3.603682] debug: unmapping init [mem 0xffffffffa5f82000-0xffffffffa6258fff] [ 3.612259] Write protecting the kernel read-only data: 28672k [ 3.615734] debug: unmapping init [mem 0xffffffffa4603000-0xffffffffa47fffff] [ 3.618819] debug: unmapping init [mem 0xffffffffa4f14000-0xffffffffa4ffffff] [ 3.654544] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.663288] systemd[1]: Detected virtualization kvm. [ 3.665455] systemd[1]: Detected architecture x86-64. [ 3.667525] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.694645] systemd[1]: No hostname configured. [ 3.696308] systemd[1]: Set hostname to . [ 3.698368] random: systemd: uninitialized urandom read (16 bytes read) [ 3.700203] systemd[1]: Initializing machine ID from random generator. [ 3.821256] random: systemd: uninitialized urandom read (16 bytes read) [ 3.824625] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.831221] random: systemd: uninitialized urandom read (16 bytes read) [ 3.834640] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.840038] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Timers. Starting Apply Kernel Variables... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Slices. [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.446258] device-mapper: uevent: version 1.0.3 [ 4.449230] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.127900] virtio_net virtio0 ens2: renamed from eth0 [ 5.156950] random: fast init done [ 5.187696] scsi host0: ata_piix [ 5.194985] scsi host1: ata_piix [ 5.196731] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.199553] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.402378] dracut-initqueue[587]: RTNETLINK answers: File exists [ 10.096498] random: crng init done [ 10.098021] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.403640] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.533230] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.816696] SELinux: Disabled at runtime. [ 11.878345] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.886238] systemd[1]: Detected virtualization kvm. [ 11.887969] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.408101] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.411430] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.415172] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.419276] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.423199] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.431169] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.435166] systemd[1]: Stopped target Switch Root. [ OK ] Stopped target Switch Root. [ OK ] Listening on udev Control Socket. [ OK ] Created slice User and Session Slice. Mounting POSIX Message Queue File System... [ OK ] Listening on udev Kernel Socket. Activating swap /dev/disk/by-label/SWAP... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Created slice system-getty.slice. [ OK ] Reached target rpc_pipefs.target. Mounting Huge Pages File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Reached target Slices. Starting Create list of required st…ce nodes for the current kernel... [ 12.498479] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on Process Core Dump Socket. [ OK ] Stopped target Initrd File Systems. Starting Remount Root and Kernel File Systems... Starting udev Coldplug all Devices... [ OK ] Stopped target Initrd Root File System. Starting Apply Kernel Variables... Mounting Kernel Debug File System... [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.875221] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.170541] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.185950] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.309073] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.320192] EDAC sbridge: Ver: 1.1.2 [ 14.865112] Key type dns_resolver registered [ 15.164481] NFS: Registering the id_resolver key type [ 15.166443] Key type id_resolver registered [ 15.169632] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. [ OK ] Started dnf makecache --timer. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg354-server login: [ 47.436655] libcfs: loading out-of-tree module taints kernel. [ 47.505378] Key type ._llcrypt registered [ 47.508213] Key type .llcrypt registered [ 47.616626] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_hostid [ 68.564559] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 70.393547] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 70.408702] alg: No test for adler32 (adler32-zlib) [ 71.929205] Lustre: Lustre: Build Version: 2.17.54_1_g695b805 [ 72.704833] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 74.480562] Key type lgssc registered [ 76.534707] Lustre: Echo OBD driver; http://www.lustre.org/ [ 95.105255] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 118.575151] hrtimer: interrupt took 2993497 ns [ 136.818455] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 148.983563] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 149.037193] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 150.314925] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 150.354168] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 150.432595] Lustre: lustre-MDT0000: new disk, initializing [ 150.524227] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 150.544631] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 154.655913] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 166.521332] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 166.613119] Lustre: 6379:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 166.644143] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 166.650711] Lustre: Skipped 1 previous similar message [ 166.753172] Lustre: lustre-MDT0001: new disk, initializing [ 166.800727] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 166.834880] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 166.842983] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 170.579933] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 175.160905] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 187.337016] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 187.790305] Lustre: lustre-OST0000: new disk, initializing [ 187.799337] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 187.812120] Lustre: 8282:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 187.928205] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 195.600505] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 195.612789] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 195.655385] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 195.664974] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 209.743582] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 209.896800] Lustre: lustre-OST0001: new disk, initializing [ 209.899764] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 209.903980] Lustre: 9337:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 209.962730] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 214.573192] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 218.645074] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 218.651638] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 218.700925] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 226.995179] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 235.789760] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 242.561309] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing check_logdir /tmp/testlogs/ [ 249.050057] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing yml_node [ 254.429960] Lustre: DEBUG MARKER: Client: 2.17.54.1 [ 257.573710] Lustre: DEBUG MARKER: MDS: 2.17.54.1 [ 259.553359] Lustre: DEBUG MARKER: OSS: 2.17.54.1 [ 261.168190] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Jun 9 05:24:54 EDT 2026 [ 274.322587] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 275.697682] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 294.882254] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 294.900627] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 294.924876] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 296.745939] Lustre: server umount lustre-MDT0000 complete [ 300.127177] LustreError: 6372:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780997135 with bad export cookie 15024803424422204350 [ 300.128970] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 300.133524] LustreError: 6372:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 300.512636] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 300.513254] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 300.529359] Lustre: Skipped 4 previous similar messages [ 300.540858] Lustre: Skipped 5 previous similar messages [ 305.633241] LustreError: 9698:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 305.634826] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 305.648333] LustreError: 9698:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 305.669250] Lustre: Skipped 1 previous similar message [ 306.592814] Lustre: server umount lustre-MDT0001 complete [ 310.324646] Lustre: server umount lustre-OST0000 complete [ 313.752557] Lustre: server umount lustre-OST0001 complete [ 319.507597] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_hostid [ 328.693809] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 365.267971] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 365.511881] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 365.542681] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 365.649223] Lustre: lustre-MDT0000: new disk, initializing [ 365.742599] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 365.760413] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 369.931576] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 381.631821] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 381.761684] Lustre: 14711:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 381.872937] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 381.885117] Lustre: Skipped 1 previous similar message [ 382.072934] Lustre: lustre-MDT0001: new disk, initializing [ 382.225698] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 382.282324] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 382.299430] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 386.515704] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 395.318649] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 399.606868] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 407.301987] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 407.658549] Lustre: lustre-OST0000: new disk, initializing [ 407.663338] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 407.669364] Lustre: 16451:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 407.767073] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 409.264872] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 409.283843] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 409.437319] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 415.431677] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 426.555629] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 433.163655] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 433.426962] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 438.313993] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 438.579531] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 445.410226] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 445.413455] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 445.416607] LustreError: Skipped 1 previous similar message [ 445.417763] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 445.427899] Lustre: Skipped 1 previous similar message [ 445.456087] Lustre: Skipped 1 previous similar message [ 446.789539] Lustre: server umount lustre-OST0000 complete [ 454.121640] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 454.135875] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 454.157239] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 465.376458] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 465.777693] Lustre: server umount lustre-MDT0000 complete [ 465.889580] LustreError: 14723:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 465.906194] LustreError: 14723:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 469.621081] LustreError: 14703:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780997304 with bad export cookie 15024803424422205883 [ 469.624839] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 469.632735] LustreError: 14703:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 470.002768] Lustre: server umount lustre-MDT0001 complete [ 475.192890] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 05:28:28 (1780997308) [ 478.562853] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 480.633899] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 05:28:33 (1780997313) [ 488.148681] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 488.447785] LustreError: 18715:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 488.532896] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 492.863412] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 493.551670] LustreError: 18716:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 498.662685] LustreError: 18715:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 500.771216] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 504.546095] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 512.039379] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 516.325421] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 523.715186] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 530.322303] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 536.764648] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 546.785588] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 546.800479] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 546.818870] Lustre: Skipped 1 previous similar message [ 546.824611] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 546.835233] Lustre: Skipped 4 previous similar messages [ 551.782759] Lustre: server umount lustre-OST0000 complete [ 557.538273] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 557.546748] LustreError: Skipped 1 previous similar message [ 557.554174] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 557.583758] Lustre: Skipped 1 previous similar message [ 564.195311] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 564.212338] Lustre: Skipped 6 previous similar messages [ 570.337473] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 570.590798] Lustre: server umount lustre-MDT0000 complete [ 573.411502] LustreError: 18716:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 573.437178] LustreError: 18716:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 574.712086] LustreError: 18697:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780997409 with bad export cookie 15024803424422206758 [ 574.712408] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 574.744911] LustreError: 18697:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 575.199948] Lustre: server umount lustre-MDT0001 complete [ 588.356108] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 591.277231] Key type lgssc unregistered [ 591.584455] LNet: 21892:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 591.594086] LNetError: 21892:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 591.616884] LNet: Removed LNI 192.168.203.154@tcp [ 592.664218] Key type .llcrypt unregistered [ 592.665771] Key type ._llcrypt unregistered [ 615.149864] Key type ._llcrypt registered [ 615.152352] Key type .llcrypt registered [ 615.303904] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 616.440182] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 616.528371] alg: No test for adler32 (adler32-zlib) [ 617.621324] Lustre: Lustre: Build Version: 2.17.54_1_g695b805 [ 617.907346] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 619.624328] Key type lgssc registered [ 620.904349] Lustre: Echo OBD driver; http://www.lustre.org/ [ 630.136806] loop: module loaded [ 630.249975] loop0: detected capacity change from 0 to 204800000 [ 630.629224] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 630.641577] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 630.986646] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 638.178560] loop0: detected capacity change from 0 to 204800000 [ 638.369142] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 638.395371] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 638.404923] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 644.151458] loop0: detected capacity change from 0 to 204800000 [ 649.385740] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 649.421701] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 650.713459] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 650.797556] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 650.900862] Lustre: lustre-MDT0000: new disk, initializing [ 651.003943] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 651.025788] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 655.313435] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 663.903821] loop1: detected capacity change from 0 to 204800000 [ 669.226817] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 669.586915] Lustre: lustre-OST07c6: new disk, initializing [ 669.591357] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 669.605519] Lustre: Skipped 1 previous similar message [ 669.612371] Lustre: 25046:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 669.709231] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 676.480701] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 679.007636] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 679.017452] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 679.062243] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 682.360993] Lustre: 24050:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 682.382646] Lustre: 24050:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 682.393757] Lustre: 24050:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 682.401474] Lustre: 24050:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 682.415616] Lustre: 24050:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 682.423381] Lustre: 24050:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 698.535300] Lustre: 24050:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 698.542482] Lustre: 24050:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 698.549074] Lustre: 24050:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 698.559696] Lustre: 24050:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 698.574090] Lustre: 24050:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 698.581893] Lustre: 24050:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 703.688391] Lustre: 24051:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 703.694303] Lustre: 24051:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 703.699030] Lustre: 24051:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 703.703223] Lustre: 24051:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 703.707968] Lustre: 24051:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 703.714486] Lustre: 24051:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 703.721878] Lustre: 24051:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 703.729311] Lustre: 24051:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 703.733657] Lustre: 24051:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 703.740330] Lustre: 24051:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 703.745122] Lustre: 24051:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 703.749717] Lustre: 24051:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 708.838361] Lustre: 24052:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 708.858082] Lustre: 24052:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 708.868520] Lustre: 24052:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 708.878177] Lustre: 24052:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 708.892719] Lustre: 24052:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 708.903504] Lustre: 24052:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 708.918258] Lustre: 24052:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 708.932237] Lustre: 24052:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 708.946526] Lustre: 24052:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 708.958275] Lustre: 24052:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 708.972709] Lustre: 24052:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 708.983903] Lustre: 24052:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 713.792921] Lustre: 25053:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 713.811343] Lustre: 25053:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 713.819012] Lustre: 25053:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 713.824914] Lustre: 25053:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 713.832987] Lustre: 25053:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 713.839698] Lustre: 25053:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 713.847855] Lustre: 25053:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 713.856861] Lustre: 25053:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 713.867230] Lustre: 25053:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 713.887020] Lustre: 25053:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 713.896534] Lustre: 25053:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 713.904119] Lustre: 25053:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 724.959751] Lustre: 24051:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 724.976678] Lustre: 24051:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 5 previous similar messages [ 724.983970] Lustre: 24051:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 724.993980] Lustre: 24051:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 725.019783] Lustre: 24051:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 725.029221] Lustre: 24051:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 725.038111] Lustre: 24051:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 725.052817] Lustre: 24051:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 725.067413] Lustre: 24051:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 725.083269] Lustre: 24051:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 725.095521] Lustre: 24051:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 725.112650] Lustre: 24051:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 774.114796] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 774.128151] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 779.234165] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 779.843823] Lustre: server umount lustre-OST07c6 complete [ 789.821899] Lustre: server umount lustre-MDT0000 complete [ 806.386994] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 809.450321] Key type lgssc unregistered [ 809.736430] LNet: 27575:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 809.746268] LNetError: 27575:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 809.762634] LNet: Removed LNI 192.168.203.154@tcp [ 810.518375] Key type .llcrypt unregistered [ 810.521255] Key type ._llcrypt unregistered [ 831.144949] Key type ._llcrypt registered [ 831.150688] Key type .llcrypt registered [ 831.266986] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 832.248202] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 832.278342] alg: No test for adler32 (adler32-zlib) [ 833.463790] Lustre: Lustre: Build Version: 2.17.54_1_g695b805 [ 833.682594] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 835.360160] Key type lgssc registered [ 836.623432] Lustre: Echo OBD driver; http://www.lustre.org/ [ 844.974787] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 845.002414] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 846.587701] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 851.014721] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 857.066732] LustreError: 28830:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 857.099000] LustreError: 28830:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 860.426373] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 860.699750] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 865.893544] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 872.022697] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 876.252876] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 882.716096] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 883.052777] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 889.049803] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 891.248549] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 894.635850] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 910.818049] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 910.835684] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 910.865714] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 911.842981] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 911.854044] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 911.859623] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 911.881654] Lustre: Skipped 1 previous similar message [ 915.011833] Lustre: server umount lustre-OST0000 complete [ 922.080994] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 922.082558] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 922.083981] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 922.098622] Lustre: Skipped 1 previous similar message [ 925.058554] Lustre: server umount lustre-MDT0000 complete [ 927.203619] LustreError: 30446:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 927.217885] LustreError: 30446:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 929.370884] LustreError: 28810:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780997764 with bad export cookie 11234562540718858416 [ 929.373639] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 929.382483] LustreError: 28810:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 929.705416] Lustre: server umount lustre-MDT0001 complete [ 940.659316] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 943.553889] Key type lgssc unregistered [ 943.938703] LNet: 32215:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 943.944713] LNetError: 32215:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 943.957898] LNet: Removed LNI 192.168.203.154@tcp [ 945.053196] Key type .llcrypt unregistered [ 945.057096] Key type ._llcrypt unregistered [ 964.845534] Key type ._llcrypt registered [ 964.849313] Key type .llcrypt registered [ 965.028983] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 966.589524] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 966.618397] alg: No test for adler32 (adler32-zlib) [ 967.712851] Lustre: Lustre: Build Version: 2.17.54_1_g695b805 [ 967.966113] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 969.704164] Key type lgssc registered [ 971.240430] Lustre: Echo OBD driver; http://www.lustre.org/ [ 980.345601] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 980.403936] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 982.153763] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 988.029977] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 991.718931] LustreError: 33472:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 991.744772] LustreError: 33472:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 996.838494] LustreError: 33473:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 998.638289] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 999.053265] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1004.089518] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 1010.586586] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1014.821630] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1022.618910] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1023.082103] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1026.171506] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1029.244527] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 1035.893719] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1047.028055] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1047.032441] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1047.064109] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1047.088605] Lustre: Skipped 1 previous similar message [ 1052.143616] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1052.155728] Lustre: Skipped 1 previous similar message [ 1052.404446] Lustre: server umount lustre-OST0000 complete [ 1057.250063] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1057.259317] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1057.261900] Lustre: Skipped 2 previous similar messages [ 1057.276094] Lustre: Skipped 1 previous similar message [ 1062.373807] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1062.383678] Lustre: Skipped 1 previous similar message [ 1063.004852] Lustre: server umount lustre-MDT0000 complete [ 1067.060777] LustreError: 33452:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780997901 with bad export cookie 1167884572135047278 [ 1067.063181] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1067.075615] LustreError: 33452:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1067.537943] Lustre: server umount lustre-MDT0001 complete [ 1078.736179] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 1081.699593] Key type lgssc unregistered [ 1081.976907] LNet: 36649:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1081.981970] LNetError: 36649:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1083.049182] LNet: Removed LNI 192.168.203.154@tcp [ 1083.951744] Key type .llcrypt unregistered [ 1083.954643] Key type ._llcrypt unregistered [ 1112.631174] Key type ._llcrypt registered [ 1112.634203] Key type .llcrypt registered [ 1112.760381] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 1113.794611] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1113.852442] alg: No test for adler32 (adler32-zlib) [ 1115.038030] Lustre: Lustre: Build Version: 2.17.54_1_g695b805 [ 1115.433624] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 1117.201778] Key type lgssc registered [ 1118.521410] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1126.961897] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1126.988945] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1128.595186] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1133.536681] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 1138.155905] LustreError: 38050:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1138.188996] LustreError: 38050:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1143.019663] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1143.265772] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 1143.432880] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1148.203050] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 1155.124601] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1159.903857] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1168.336723] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1168.810846] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1173.941056] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1175.655232] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 1181.856924] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1193.447717] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1193.455744] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1193.472682] Lustre: Skipped 2 previous similar messages [ 1193.483905] Lustre: Skipped 1 previous similar message [ 1197.556928] Lustre: server umount lustre-MDT0000 complete [ 1198.567772] LustreError: 39919:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1198.593912] LustreError: 39919:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1201.770929] LustreError: 38029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780998036 with bad export cookie 7981901298484364703 [ 1201.779209] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1201.792426] LustreError: 38029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1203.689574] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1203.689574] LustreError: 39919:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1203.689586] LustreError: 39919:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1203.703352] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1203.741483] Lustre: Skipped 2 previous similar messages [ 1206.689615] LustreError: 38045:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1206.690096] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1206.698151] LustreError: 38045:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1206.709479] Lustre: Skipped 1 previous similar message [ 1208.314143] Lustre: server umount lustre-MDT0001 complete [ 1224.460655] Lustre: server umount lustre-OST0000 complete [ 1244.122544] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 1247.452941] Key type lgssc unregistered [ 1247.814597] LNet: 41553:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1247.828618] LNetError: 41553:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1247.848859] LNet: Removed LNI 192.168.203.154@tcp [ 1248.888237] Key type .llcrypt unregistered [ 1248.892056] Key type ._llcrypt unregistered [ 1277.568586] Key type ._llcrypt registered [ 1277.570447] Key type .llcrypt registered [ 1277.703650] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 1278.685397] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1278.760858] alg: No test for adler32 (adler32-zlib) [ 1280.041675] Lustre: Lustre: Build Version: 2.17.54_1_g695b805 [ 1280.448881] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 1282.241082] Key type lgssc registered [ 1283.353281] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1293.189188] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1293.225979] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1294.680956] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1299.617973] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 1305.066067] LustreError: 42964:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1305.090132] LustreError: 42964:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1309.126516] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1309.603189] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1314.478517] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 1320.561098] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1325.277637] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1333.367790] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1333.903648] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1340.098096] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1340.739608] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 1347.975306] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1355.330328] Lustre: lustre-OST0000: Not available for connect from 192.168.203.54@tcp (stopping) [ 1355.747522] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1355.747749] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1355.761051] LustreError: Skipped 1 previous similar message [ 1355.790214] Lustre: Skipped 1 previous similar message [ 1360.392125] Lustre: lustre-OST0000: Not available for connect from 192.168.203.54@tcp (stopping) [ 1360.399870] Lustre: Skipped 2 previous similar messages [ 1360.847569] Lustre: server umount lustre-OST0000 complete [ 1380.322953] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1380.328486] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1380.341622] Lustre: Skipped 1 previous similar message [ 1380.349841] Lustre: Skipped 1 previous similar message [ 1385.441786] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1385.450180] Lustre: Skipped 1 previous similar message [ 1385.794014] Lustre: server umount lustre-MDT0000 complete [ 1389.821149] LustreError: 42945:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780998224 with bad export cookie 6530483226827567058 [ 1389.825831] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1389.830290] LustreError: 42945:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1404.384231] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1404.638653] Lustre: server umount lustre-MDT0001 complete [ 1418.588670] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 1421.710345] Key type lgssc unregistered [ 1422.146873] LNet: 46318:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1422.161623] LNetError: 46318:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1422.208666] LNet: Removed LNI 192.168.203.154@tcp [ 1423.114503] Key type .llcrypt unregistered [ 1423.118160] Key type ._llcrypt unregistered [ 1451.507257] Key type ._llcrypt registered [ 1451.510188] Key type .llcrypt registered [ 1451.635802] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 1452.810463] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1452.850038] alg: No test for adler32 (adler32-zlib) [ 1454.167421] Lustre: Lustre: Build Version: 2.17.54_1_g695b805 [ 1454.383379] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 1456.057104] Key type lgssc registered [ 1457.435650] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1466.307451] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1466.350136] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1467.926682] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1473.086107] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 1478.123959] LustreError: 47719:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1478.137930] LustreError: 47719:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1483.236982] LustreError: 47720:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1483.267177] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1483.792040] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1488.879783] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 1495.411087] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1499.630198] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1506.764500] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1507.196995] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1513.238994] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 1513.344820] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1520.162535] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1526.450074] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1530.386595] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1543.096907] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-06-09 5h46m15s [ 1545.093627] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1547.869408] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.54@tcp (stopping) [ 1548.939103] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.54@tcp (stopping) [ 1549.282236] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1549.299576] Lustre: Skipped 1 previous similar message [ 1550.372145] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1550.381804] Lustre: Skipped 5 previous similar messages [ 1553.869519] Lustre: server umount lustre-MDT0000 complete [ 1553.996355] LustreError: 48829:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1556.463649] LustreError: 47719:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1556.489813] LustreError: 47719:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1558.001133] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1562.610213] LustreError: 47720:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1562.652291] LustreError: 47720:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1566.129551] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1566.407409] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1566.735224] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1566.809123] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1567.800211] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1571.731971] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 1575.573784] Lustre: DEBUG MARKER: Wait for df (35996) ... [ 1577.445385] Lustre: DEBUG MARKER: done [ 1585.142469] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1585.150505] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1585.164840] Lustre: Skipped 1 previous similar message [ 1585.174766] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1587.178569] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1587.181266] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1590.319916] Lustre: server umount lustre-OST0000 complete [ 1598.437994] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1598.469394] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1598.472025] Lustre: Skipped 4 previous similar messages [ 1610.208125] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1610.530249] Lustre: server umount lustre-MDT0000 complete [ 1611.758679] LustreError: 48829:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1611.793545] LustreError: 48829:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1614.764055] LustreError: 47699:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780998449 with bad export cookie 1863373982605362077 [ 1614.766258] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1614.787504] LustreError: 47699:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1615.166870] Lustre: server umount lustre-MDT0001 complete [ 1627.852303] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 1630.951358] Key type lgssc unregistered [ 1631.214072] LNet: 52273:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1631.218900] LNetError: 52273:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1631.232485] LNet: Removed LNI 192.168.203.154@tcp [ 1632.351826] Key type .llcrypt unregistered [ 1632.356320] Key type ._llcrypt unregistered [ 1680.624445] Key type ._llcrypt registered [ 1680.631879] Key type .llcrypt registered [ 1680.793828] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 1682.473000] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1682.498928] alg: No test for adler32 (adler32-zlib) [ 1683.586729] Lustre: Lustre: Build Version: 2.17.54_1_g695b805 [ 1683.853519] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 1685.560706] Key type lgssc registered [ 1686.837107] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1695.925654] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1695.947855] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1697.369182] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1697.408881] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1697.673221] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1702.388440] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 1711.691484] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1711.763905] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1711.811492] Lustre: 54464:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1712.031516] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1716.700434] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 1723.586541] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1728.811602] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1737.119980] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1737.494765] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1737.796178] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1744.815441] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 1747.230506] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1753.472455] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1761.264969] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-06-09 5h49m53s [ 1763.296378] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1772.745055] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1775.405278] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1775.477222] LustreError: 56656:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1795.658869] Lustre: lustre-MDT0000: Client 8fb5e109-509d-41d9-957c-0c58aaac8fa8 (at 192.168.203.54@tcp) reconnecting [ 1815.477645] LustreError: 56656:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1817.826912] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1820.480395] Lustre: DEBUG MARKER: done [ 1829.347997] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1829.348508] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1829.360555] LustreError: Skipped 1 previous similar message [ 1829.361653] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1829.394605] Lustre: Skipped 1 previous similar message [ 1833.551311] Lustre: server umount lustre-OST0000 complete [ 1839.587112] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1839.590811] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1839.614861] Lustre: Skipped 1 previous similar message [ 1839.642573] Lustre: Skipped 2 previous similar messages [ 1840.674393] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1840.684535] Lustre: Skipped 1 previous similar message [ 1843.563631] Lustre: server umount lustre-MDT0000 complete [ 1847.840605] LustreError: 55201:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780998682 with bad export cookie 6263999689526024650 [ 1847.852570] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1847.860611] LustreError: 55201:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1848.281306] Lustre: server umount lustre-MDT0001 complete [ 1859.492286] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 1861.886717] Key type lgssc unregistered [ 1862.206391] LNet: 58441:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1862.232951] LNetError: 58441:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1862.251937] LNet: Removed LNI 192.168.203.154@tcp [ 1863.121526] Key type .llcrypt unregistered [ 1863.124226] Key type ._llcrypt unregistered [ 1913.984454] Key type ._llcrypt registered [ 1913.988181] Key type .llcrypt registered [ 1914.065062] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 1915.286441] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1915.427821] alg: No test for adler32 (adler32-zlib) [ 1916.607053] Lustre: Lustre: Build Version: 2.17.54_1_g695b805 [ 1916.853387] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 1918.536206] Key type lgssc registered [ 1919.794613] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1923.634956] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1923.672556] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1925.011176] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1925.061415] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1925.271083] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1926.301607] Lustre: Failing over lustre-MDT0000 [ 1926.652974] Lustre: server umount lustre-MDT0000 complete [ 1934.057634] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 05:52:46 (1780998766) [ 1942.569878] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1943.276871] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1949.272568] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 1959.256027] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1959.502197] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1959.634656] Lustre: 61221:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1959.886106] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1964.010880] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 1970.940568] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1974.973353] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1983.056789] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1983.332541] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1983.645581] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1990.603414] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 1994.008190] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1997.360921] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2002.949433] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2011.106388] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2011.113613] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2011.128449] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2014.693429] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2014.696134] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2014.700122] Lustre: Skipped 1 previous similar message [ 2014.711989] Lustre: Skipped 1 previous similar message [ 2017.100463] Lustre: server umount lustre-MDT0000 complete [ 2021.112441] LustreError: 61214:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780998856 with bad export cookie 2869698617794672464 [ 2021.120068] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2021.122664] LustreError: 61214:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2021.408665] Lustre: server umount lustre-MDT0001 complete [ 2025.364087] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2031.492626] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2040.288462] Lustre: 60145:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780998859/real 1780998859] req@000000000121095c x1867512546369024/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1780998875 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2040.329920] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2041.844676] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2042.338388] Lustre: 60146:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780998861/real 1780998861] req@00000000c8317651 x1867512546369152/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1780998877 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2045.408288] Lustre: 60147:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780998864/real 1780998864] req@0000000039bb7262 x1867512546369280/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1780998880 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2047.530912] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x27d337b04614c033 [ 2047.570447] Lustre: MGC192.168.203.154@tcp: Connection restored to 0@lo (at 0@lo) [ 2047.899670] LustreError: 62848:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2047.921367] LustreError: 62848:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2048.021128] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2048.104748] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2049.059089] Lustre: 60148:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780998867/real 1780998867] req@000000000da401f9 x1867512546369536/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1780998883 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2049.084081] LustreError: 62849:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2049.126170] LustreError: 62849:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2051.109036] LustreError: 64659:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2051.145305] LustreError: 64659:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2052.718410] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 2053.141120] Lustre: 60146:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780998872/real 1780998872] req@00000000738039a8 x1867512546369664/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1780998888 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2056.169799] LustreError: 64658:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2056.188517] LustreError: 64658:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2061.290610] LustreError: 64660:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2061.324228] LustreError: 64660:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2062.777476] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2063.156100] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2068.364815] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 2068.479843] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2069.494419] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2075.846553] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2081.095771] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2089.968968] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2089.979168] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2089.989092] Lustre: Skipped 1 previous similar message [ 2090.019532] Lustre: Skipped 3 previous similar messages [ 2095.075552] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2095.320788] Lustre: server umount lustre-MDT0000 complete [ 2099.699445] LustreError: 61944:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780998934 with bad export cookie 2869698617794674739 [ 2099.700254] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2099.712244] LustreError: 61944:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2100.199396] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2100.200158] LustreError: 64659:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2100.214607] Lustre: Skipped 1 previous similar message [ 2100.258619] LustreError: 64659:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2100.363475] Lustre: server umount lustre-MDT0001 complete [ 2108.585218] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2118.608110] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2124.705263] LustreError: 60144:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000000f918438 x1867512546398976/t0(0) o250->MGC192.168.203.154@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2125.166333] LustreError: 62849:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2125.295655] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2125.383279] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2129.384899] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2130.617520] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 2141.082444] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2146.803859] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2148.154879] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 2155.886529] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2161.111777] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2172.387668] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2172.403015] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2172.404270] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2172.405888] Lustre: Skipped 2 previous similar messages [ 2172.434539] Lustre: Skipped 4 previous similar messages [ 2176.655278] Lustre: server umount lustre-MDT0000 complete [ 2177.509323] LustreError: 67315:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2177.553597] LustreError: 67315:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 2180.954224] LustreError: 61214:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780999015 with bad export cookie 2869698617794676118 [ 2180.957485] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2180.963556] LustreError: 61214:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2181.347023] Lustre: server umount lustre-MDT0001 complete [ 2188.874542] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2193.070777] Lustre: DEBUG MARKER: files compared the same [ 2206.011558] Lustre: server umount lustre-OST0000 complete [ 2223.778915] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 2226.857533] Key type lgssc unregistered [ 2227.211956] LNet: 70150:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2227.229121] LNetError: 70150:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2227.241692] LNet: Removed LNI 192.168.203.154@tcp [ 2228.191193] Key type .llcrypt unregistered [ 2228.195472] Key type ._llcrypt unregistered [ 2253.850331] Key type ._llcrypt registered [ 2253.854892] Key type .llcrypt registered [ 2254.043728] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 2255.883962] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2255.928269] alg: No test for adler32 (adler32-zlib) [ 2257.115943] Lustre: Lustre: Build Version: 2.17.54_1_g695b805 [ 2257.483881] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 2259.202000] Key type lgssc registered [ 2260.642579] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2269.940823] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2269.982221] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2271.632703] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2276.618667] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug +malloc all [ 2281.953847] LustreError: 71556:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2282.007517] LustreError: 71556:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2286.324359] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2286.627214] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2292.100707] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug +malloc all [ 2298.514649] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2303.368142] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2311.862813] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2312.247277] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2316.350397] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2320.341241] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug +malloc all [ 2327.983197] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2341.857721] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2341.860180] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2341.864309] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2341.887094] Lustre: Skipped 1 previous similar message [ 2346.979579] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2346.995977] Lustre: Skipped 2 previous similar messages [ 2347.524498] Lustre: server umount lustre-OST0000 complete [ 2352.097650] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2352.120679] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2352.132265] Lustre: Skipped 1 previous similar message [ 2357.223653] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2357.236539] Lustre: Skipped 1 previous similar message [ 2358.132440] Lustre: server umount lustre-MDT0000 complete [ 2362.338685] LustreError: 71552:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2362.366125] LustreError: 71552:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2363.317341] LustreError: 71537:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780999198 with bad export cookie 3113171816491384358 [ 2363.336868] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2363.350862] LustreError: 71537:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2363.718929] Lustre: server umount lustre-MDT0001 complete [ 2377.852596] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 2381.363514] Key type lgssc unregistered [ 2381.632677] LNet: 74755:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2381.638534] LNetError: 74755:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2382.706498] LNet: Removed LNI 192.168.203.154@tcp [ 2383.472873] Key type .llcrypt unregistered [ 2383.475521] Key type ._llcrypt unregistered [ 2408.022575] Key type ._llcrypt registered [ 2408.024879] Key type .llcrypt registered [ 2408.146666] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 2409.873094] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2409.971779] alg: No test for adler32 (adler32-zlib) [ 2411.313268] Lustre: Lustre: Build Version: 2.17.54_1_g695b805 [ 2411.632075] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 2413.336429] Key type lgssc registered [ 2414.563972] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2424.307571] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2424.338556] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2439.972485] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2445.088280] LustreError: 76128:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.154@tcp: failed processing log, type 4: rc = -110 [ 2471.776894] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2478.920829] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 2486.883681] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2496.822828] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2497.030808] Lustre: *** cfs_fail_loc=706, val=0*** [ 2497.545580] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2497.622465] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2502.550791] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 2502.629096] LustreError: 76153:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2502.649896] LustreError: 76153:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2506.733715] LustreError: 76154:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2506.762993] LustreError: 76154:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2510.920265] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2511.270050] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2516.067771] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 2522.475986] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2526.309187] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2531.814876] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2531.822129] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2531.822434] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2531.841491] Lustre: Skipped 1 previous similar message [ 2534.786980] Lustre: server umount lustre-OST0000 complete [ 2542.050976] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2542.051441] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2542.052452] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2542.052457] Lustre: Skipped 1 previous similar message [ 2542.075256] Lustre: Skipped 1 previous similar message [ 2547.178944] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2547.186288] Lustre: Skipped 2 previous similar messages [ 2551.273874] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2551.293664] Lustre: Skipped 1 previous similar message [ 2553.313773] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2553.630964] Lustre: server umount lustre-MDT0000 complete [ 2556.390676] LustreError: 77112:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2556.422233] LustreError: 77112:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2557.725709] LustreError: 76136:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780999392 with bad export cookie 350258849414661448 [ 2557.725972] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2557.748625] LustreError: 76136:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2558.246353] Lustre: server umount lustre-MDT0001 complete [ 2570.166191] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 2573.050764] Key type lgssc unregistered [ 2573.327846] LNet: 79402:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2573.338059] LNetError: 79402:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2573.366300] LNet: Removed LNI 192.168.203.154@tcp [ 2574.104210] Key type .llcrypt unregistered [ 2574.106580] Key type ._llcrypt unregistered [ 2602.531854] Key type ._llcrypt registered [ 2602.535747] Key type .llcrypt registered [ 2602.670671] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 2603.905028] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2603.932814] alg: No test for adler32 (adler32-zlib) [ 2604.982428] Lustre: Lustre: Build Version: 2.17.54_1_g695b805 [ 2605.157298] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 2606.776288] Key type lgssc registered [ 2607.573729] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2615.835657] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2615.853802] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2620.602195] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 2628.743121] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2629.718831] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2634.726252] LustreError: 81477:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2634.750551] LustreError: 81477:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2635.050707] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 2639.848193] LustreError: 81480:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2639.863431] LustreError: 81480:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2644.704259] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2644.963146] LustreError: 81481:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2644.990280] LustreError: 81481:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2645.198284] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2650.604231] LustreError: 81476:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2650.633474] LustreError: 81476:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 2652.234136] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 2654.763067] LustreError: 81477:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2654.803630] LustreError: 81477:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2661.719459] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2661.738143] Lustre: MGS: Client 9e124209-5b8b-41d4-be9c-0615ab6f1b88 (at 0@lo) reconnecting [ 2661.766043] LustreError: 81491:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2662.067964] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2662.183407] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2667.315497] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 2682.859114] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2682.865388] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2682.876262] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2682.897055] Lustre: Skipped 1 previous similar message [ 2687.287629] Lustre: server umount lustre-OST0000 complete [ 2693.090450] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2693.090685] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2693.099776] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2693.117526] Lustre: Skipped 2 previous similar messages [ 2697.331139] Lustre: server umount lustre-MDT0000 complete [ 2698.211356] LustreError: 81480:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2698.224461] LustreError: 81480:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2701.575698] Lustre: server umount lustre-MDT0001 complete [ 2705.527890] Lustre: server umount MGS complete [ 2716.557369] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 06:05:49 (1780999549) [ 2743.575232] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_hostid [ 2752.883461] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 2791.588028] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2796.366700] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 2806.264074] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2806.580919] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2806.637434] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2806.740472] Lustre: lustre-MDT0001: new disk, initializing [ 2806.867927] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2806.915920] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2806.936424] Lustre: 88546:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2806.956125] LustreError: 88546:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2812.068517] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 2823.931926] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2824.179916] Lustre: lustre-OST0000: new disk, initializing [ 2824.186182] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2824.194054] Lustre: 89295:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2824.298043] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2831.216254] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 2834.017438] Lustre: 89721:0:(client.c:1638:after_reply()) @@@ resending request on EINPROGRESS req@00000000204be1ee x1867513268839936/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1780999684 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2839.127797] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2845.869130] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2845.882830] Lustre: MGS: Client 7f2e8e58-5482-4d7b-b0b4-5738da485a14 (at 0@lo) reconnecting [ 2845.936174] Lustre: 90228:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2845.947513] LustreError: 88532:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2845.975069] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2846.050267] Lustre: lustre-MDT0000: new disk, initializing [ 2846.228777] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2846.261393] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2846.425644] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2847.335014] Lustre: 90252:0:(client.c:1638:after_reply()) @@@ resending request on EINPROGRESS req@000000001217df14 x1867513268846464/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1780999698 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2850.717983] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 2852.403382] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2852.430507] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2852.558166] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2877.932347] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2877.937752] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2877.959742] Lustre: Skipped 1 previous similar message [ 2877.976714] Lustre: Skipped 2 previous similar messages [ 2881.846284] Lustre: server umount lustre-OST0000 complete [ 2888.163205] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2888.165793] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2888.177881] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2891.829224] Lustre: server umount lustre-MDT0000 complete [ 2893.284143] LustreError: 88518:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2893.298201] LustreError: 88518:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2901.801391] Lustre: server umount lustre-MDT0001 complete [ 2905.473730] Lustre: server umount MGS complete [ 2916.410744] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 06:09:09 (1780999749) [ 2935.005519] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 2938.186650] Key type lgssc unregistered [ 2938.484236] LNet: 92714:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2938.494623] LNetError: 92714:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2939.558950] LNet: Removed LNI 192.168.203.154@tcp [ 2940.308577] Key type .llcrypt unregistered [ 2940.311050] Key type ._llcrypt unregistered [ 2959.639721] Key type ._llcrypt registered [ 2959.644205] Key type .llcrypt registered [ 2959.745555] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 2961.417525] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2961.505673] alg: No test for adler32 (adler32-zlib) [ 2962.562867] Lustre: Lustre: Build Version: 2.17.54_1_g695b805 [ 2962.863826] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 2964.544317] Key type lgssc registered [ 2965.854853] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2974.150989] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2974.188440] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2975.800155] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2980.485499] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 2985.964607] LustreError: 93921:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2985.975102] LustreError: 93921:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2989.595888] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2989.950250] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2994.194497] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 2999.831298] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3004.886520] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3012.220254] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3012.709821] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3019.168613] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 3020.915121] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 3026.213334] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3035.620441] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3035.632828] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3035.675171] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3036.646404] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3036.648692] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3041.549400] Lustre: server umount lustre-OST0000 complete [ 3046.371048] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3046.388044] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3046.410870] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3046.422065] Lustre: Skipped 1 previous similar message [ 3052.005376] LustreError: 93922:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3052.026064] LustreError: 93922:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3052.052976] Lustre: server umount lustre-MDT0000 complete [ 3056.004815] LustreError: 93901:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780999890 with bad export cookie 9905861470557423804 [ 3056.011206] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3056.022482] LustreError: 93901:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3056.409773] Lustre: server umount lustre-MDT0001 complete [ 3067.875573] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 3070.981930] Key type lgssc unregistered [ 3071.301165] LNet: 97101:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3071.309585] LNetError: 97101:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3072.360185] LNet: Removed LNI 192.168.203.154@tcp [ 3073.214683] Key type .llcrypt unregistered [ 3073.219218] Key type ._llcrypt unregistered [ 3091.889339] Key type ._llcrypt registered [ 3091.892451] Key type .llcrypt registered [ 3092.058596] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 3093.210279] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3093.220399] alg: No test for adler32 (adler32-zlib) [ 3094.556142] Lustre: Lustre: Build Version: 2.17.54_1_g695b805 [ 3094.846377] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 3096.656613] Key type lgssc registered [ 3097.958226] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3104.640632] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3104.645935] LustreError: 98160:0:(tgt_mount.c:2522:server_fill_super()) cfs_race id 716 sleeping [ 3104.693707] LustreError: 98167:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 waking [ 3104.699131] LustreError: 98160:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 awake: rc=4951 [ 3104.710078] LustreError: 98160:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3104.718794] LustreError: 98160:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3104.738443] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3106.552633] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3115.457525] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3115.847714] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3120.451313] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 3126.904465] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing lsmod [ 3130.484878] LustreError: 99343:0:(tgt_mount.c:2522:server_fill_super()) cfs_race id 716 sleeping [ 3130.497065] LustreError: 99344:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 waking [ 3130.509130] LustreError: 99343:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 awake: rc=4988 [ 3130.519168] LustreError: 99343:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3130.527537] LustreError: 99343:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3130.540350] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3131.054774] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3132.124622] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3136.488838] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3136.494803] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3136.504911] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3137.509298] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3137.509702] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3137.531164] Lustre: Skipped 2 previous similar messages [ 3140.336596] Lustre: server umount lustre-MDT0000 complete [ 3140.579777] LustreError: 98691:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3140.596173] LustreError: 98691:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 3143.844520] LustreError: 98185:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780999978 with bad export cookie 14193908980610893638 [ 3143.854756] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3143.868357] LustreError: 98185:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3144.253454] Lustre: server umount lustre-MDT0001 complete [ 3159.813903] Lustre: server umount lustre-OST0000 complete [ 3170.568170] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3171.230653] LustreError: 100410:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3171.386033] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3176.419838] LustreError: 100411:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3176.535566] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 3181.540820] LustreError: 100410:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3186.167358] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3186.617903] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3191.274145] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 3199.428624] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3204.701141] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3214.307382] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3214.838832] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3216.902552] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:97) [ 3221.619292] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 3227.956235] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3242.466086] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3242.492439] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3242.509091] Lustre: Skipped 1 previous similar message [ 3242.512211] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3246.561948] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3246.567515] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3246.575162] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3246.630352] Lustre: Skipped 1 previous similar message [ 3247.959523] Lustre: server umount lustre-OST0000 complete [ 3253.217845] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3253.232407] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3253.271706] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3259.686623] Lustre: server umount lustre-MDT0000 complete [ 3261.956326] LustreError: 100410:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3261.996883] LustreError: 100410:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3264.567207] LustreError: 100390:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781000099 with bad export cookie 14193908980610894583 [ 3264.587966] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3264.589077] LustreError: 100390:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3265.176845] Lustre: server umount lustre-MDT0001 complete [ 3278.426375] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 3282.058276] Key type lgssc unregistered [ 3282.437634] LNet: 103620:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3282.449089] LNetError: 103620:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3282.471077] LNet: Removed LNI 192.168.203.154@tcp [ 3283.472905] Key type .llcrypt unregistered [ 3283.475519] Key type ._llcrypt unregistered [ 3311.892552] Key type ._llcrypt registered [ 3311.898447] Key type .llcrypt registered [ 3312.055708] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 3313.715057] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3313.787932] alg: No test for adler32 (adler32-zlib) [ 3314.863577] Lustre: Lustre: Build Version: 2.17.54_1_g695b805 [ 3315.140649] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 3316.848213] Key type lgssc registered [ 3318.660950] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3328.806403] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3328.824943] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3330.369061] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3335.538776] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 3340.795242] LustreError: 105024:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3340.825535] LustreError: 105024:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3345.897312] LustreError: 105023:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3346.990883] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3347.238504] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3352.376270] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 3358.814106] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3362.901829] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3369.409308] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3369.806133] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3375.093340] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 3375.993394] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3383.228153] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3389.293739] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3395.036703] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3401.697960] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3401.704581] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3401.708467] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3401.711092] Lustre: Skipped 1 previous similar message [ 3401.724733] Lustre: Skipped 1 previous similar message [ 3405.014814] Lustre: server umount lustre-OST0000 complete [ 3411.940989] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3411.973333] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3411.982252] Lustre: Skipped 1 previous similar message [ 3415.369282] Lustre: server umount lustre-MDT0000 complete [ 3417.058412] LustreError: 105023:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3417.074460] LustreError: 105023:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3419.338634] LustreError: 105005:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781000254 with bad export cookie 6997585932561637518 [ 3419.340578] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3419.359751] LustreError: 105005:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3419.614859] Lustre: server umount lustre-MDT0001 complete [ 3430.343595] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 3433.201256] Key type lgssc unregistered [ 3433.522597] LNet: 108304:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3433.533698] LNetError: 108304:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3433.547722] LNet: Removed LNI 192.168.203.154@tcp [ 3434.423236] Key type .llcrypt unregistered [ 3434.425829] Key type ._llcrypt unregistered [ 3454.068696] Key type ._llcrypt registered [ 3454.071118] Key type .llcrypt registered [ 3454.201349] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 3455.338512] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3455.369748] alg: No test for adler32 (adler32-zlib) [ 3456.652469] Lustre: Lustre: Build Version: 2.17.54_1_g695b805 [ 3456.878301] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 3458.536178] Key type lgssc registered [ 3459.719304] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3467.725768] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3467.738705] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3469.121601] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3473.283456] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 3478.505040] LustreError: 109510:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3478.526026] LustreError: 109510:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3482.294397] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3482.550745] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3486.913783] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 3494.285721] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3499.494745] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3507.671106] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3508.235352] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3513.389720] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3515.851247] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 3522.063362] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3533.794371] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3533.799866] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3533.812796] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3537.892201] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3537.899059] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3537.902678] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3537.919413] Lustre: Skipped 1 previous similar message [ 3539.793425] Lustre: server umount lustre-OST0000 complete [ 3544.033891] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3544.042407] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3544.058436] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3548.130167] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3548.130931] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3550.069650] Lustre: server umount lustre-MDT0000 complete [ 3553.254096] LustreError: 109506:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3553.283161] LustreError: 109506:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3554.389047] LustreError: 109493:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781000389 with bad export cookie 6458896782875757006 [ 3554.395955] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3554.421393] LustreError: 109493:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3554.788977] Lustre: server umount lustre-MDT0001 complete [ 3566.714507] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 3569.173636] Key type lgssc unregistered [ 3569.531083] LNet: 112687:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3569.542265] LNetError: 112687:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3569.581705] LNet: Removed LNI 192.168.203.154@tcp [ 3570.437196] Key type .llcrypt unregistered [ 3570.440093] Key type ._llcrypt unregistered [ 3597.987761] Key type ._llcrypt registered [ 3597.990189] Key type .llcrypt registered [ 3598.054346] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 3598.573499] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3598.650382] alg: No test for adler32 (adler32-zlib) [ 3599.706654] Lustre: Lustre: Build Version: 2.17.54_1_g695b805 [ 3600.027975] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 3601.729278] Key type lgssc registered [ 3603.126677] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3611.603236] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3611.623986] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3613.059153] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3616.634026] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 3622.380749] LustreError: 114088:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3622.415515] LustreError: 114088:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3627.518801] LustreError: 114089:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3627.950527] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3628.626710] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3633.340188] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 3642.037665] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3646.940526] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3654.017317] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3654.499710] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3657.582904] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:163 to 0x280000400:193) [ 3660.815647] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 3666.122179] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3682.143427] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3686.405947] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3686.415948] Lustre: Skipped 2 previous similar messages [ 3689.111392] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3689.167310] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3691.003167] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3691.016243] Lustre: Skipped 2 previous similar messages [ 3694.589081] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3694.603352] Lustre: Skipped 1 previous similar message [ 3701.792623] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.203.54@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3701.805978] Lustre: Skipped 2 previous similar messages [ 3703.061769] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.203.54@tcp 192.168.0.[2,10]@tcp [ 3703.085547] Lustre: Skipped 1 previous similar message [ 3707.805371] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.203.54/24@tcp in log lustre-MDT0000 [ 3707.812511] Lustre: Skipped 2 previous similar messages [ 3709.213877] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.54/24@tcp [ 3717.097126] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3717.102580] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3717.114037] LustreError: Skipped 1 previous similar message [ 3717.129914] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3717.363613] Lustre: server umount lustre-OST0000 complete [ 3720.673770] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3720.682737] LustreError: Skipped 4 previous similar messages [ 3720.688188] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3720.708076] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3725.286838] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3725.288978] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3725.310671] Lustre: Skipped 1 previous similar message [ 3726.750819] Lustre: server umount lustre-MDT0000 complete [ 3730.402960] LustreError: 114083:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3730.434459] LustreError: 114083:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3730.653542] LustreError: 114807:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781000565 with bad export cookie 11764522879428761122 [ 3730.666730] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3730.671189] LustreError: 114807:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3740.310337] Lustre: server umount lustre-MDT0001 complete [ 3752.674728] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 3755.553191] Key type lgssc unregistered [ 3755.873257] LNet: 118509:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3755.883481] LNetError: 118509:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3755.896221] LNet: Removed LNI 192.168.203.154@tcp [ 3756.700536] Key type .llcrypt unregistered [ 3756.702774] Key type ._llcrypt unregistered [ 3787.963576] Key type ._llcrypt registered [ 3787.965920] Key type .llcrypt registered [ 3788.070688] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 3789.131878] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3789.324466] alg: No test for adler32 (adler32-zlib) [ 3790.460330] Lustre: Lustre: Build Version: 2.17.54_1_g695b805 [ 3790.791950] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 3792.488530] Key type lgssc registered [ 3793.980742] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3802.645841] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3802.660790] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3804.182859] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3804.217812] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.203.54/24@tcp [ 3804.241517] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3809.250956] LustreError: 120060:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3809.270084] LustreError: 120060:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3809.303170] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 3814.379157] LustreError: 120059:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3818.700475] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3819.007439] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3819.043658] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.54/24@tcp [ 3819.085365] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3824.178724] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 3830.434729] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3834.077864] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3840.760886] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3841.276833] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3848.203871] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 3849.558094] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:201 to 0x280000400:225) [ 3854.541693] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3865.059134] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3865.074889] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3865.080656] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3865.093814] Lustre: Skipped 1 previous similar message [ 3865.103955] Lustre: Skipped 1 previous similar message [ 3870.187713] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3870.198069] Lustre: Skipped 1 previous similar message [ 3875.301697] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3875.306169] Lustre: Skipped 1 previous similar message [ 3879.392170] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3879.674492] Lustre: server umount lustre-OST0000 complete [ 3885.544510] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3885.544846] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3885.574589] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3890.082040] Lustre: server umount lustre-MDT0000 complete [ 3890.668466] LustreError: 121767:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3890.694766] LustreError: 121767:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3894.571617] LustreError: 120040:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781000729 with bad export cookie 838937503461015922 [ 3894.576966] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3894.584928] LustreError: 120040:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3895.423119] Lustre: server umount lustre-MDT0001 complete [ 3908.415959] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 3911.617802] Key type lgssc unregistered [ 3912.094531] LNet: 123294:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3912.111737] LNetError: 123294:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3913.128422] LNet: Removed LNI 192.168.203.154@tcp [ 3914.106744] Key type .llcrypt unregistered [ 3914.112591] Key type ._llcrypt unregistered [ 3945.586353] Key type ._llcrypt registered [ 3945.594136] Key type .llcrypt registered [ 3945.718423] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_hostid [ 3962.447626] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 3963.629315] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3963.780476] alg: No test for adler32 (adler32-zlib) [ 3964.957223] Lustre: Lustre: Build Version: 2.17.54_1_g695b805 [ 3965.148809] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 3966.842295] Key type lgssc registered [ 3968.287580] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4011.228286] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4011.279425] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4012.620120] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4012.678563] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4012.811842] Lustre: lustre-MDT0000: new disk, initializing [ 4012.987266] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4013.030600] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4018.020231] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 4029.742610] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4029.965054] Lustre: 126662:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4030.052782] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4030.060847] Lustre: Skipped 1 previous similar message [ 4030.164384] Lustre: lustre-MDT0001: new disk, initializing [ 4030.286799] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4030.332428] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4030.348052] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4035.532786] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 4044.580710] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4049.869777] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4057.178634] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4057.603724] Lustre: lustre-OST0000: new disk, initializing [ 4057.608890] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4057.627454] Lustre: 128405:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4057.724486] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4063.507716] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 4065.851127] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4065.858812] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4065.991436] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4071.305513] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4075.689876] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4075.907777] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4080.602335] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4080.813464] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4086.761100] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4086.761359] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4086.767785] LustreError: Skipped 1 previous similar message [ 4086.768768] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4086.790555] Lustre: Skipped 1 previous similar message [ 4088.717325] Lustre: server umount lustre-OST0000 complete [ 4096.996649] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4097.000497] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4097.027787] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4097.037021] Lustre: Skipped 1 previous similar message [ 4097.046572] Lustre: Skipped 1 previous similar message [ 4098.331694] Lustre: server umount lustre-MDT0000 complete [ 4102.058390] LustreError: 126655:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781000936 with bad export cookie 227212006959538810 [ 4102.063468] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4102.069227] LustreError: 126655:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4102.485762] Lustre: server umount lustre-MDT0001 complete [ 4111.303795] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4111.830611] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4116.839303] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 4116.962644] LustreError: 130476:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4116.996695] LustreError: 130476:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4122.083382] LustreError: 130475:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4126.526940] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4127.259212] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4132.581255] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 4139.287652] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4143.590932] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4151.076479] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4151.487705] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4158.006898] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 4167.496432] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4173.805540] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4174.159217] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4179.233285] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4179.503108] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4191.140661] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4191.461295] Lustre: lustre-OST0001: new disk, initializing [ 4191.466746] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4191.483053] Lustre: 133474:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4191.693082] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4200.052024] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4200.057715] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4200.182664] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4200.363550] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 4210.498264] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4210.725201] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4215.372782] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4215.563415] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4220.003899] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9fc947e39000.ost_server_uuid 50 [ 4222.268691] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9fc947e39000.ost_server_uuid in IDLE state after 0 sec [ 4236.263485] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4236.268625] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4236.268668] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4236.288428] Lustre: Skipped 1 previous similar message [ 4236.325122] Lustre: Skipped 1 previous similar message [ 4238.582711] Lustre: server umount lustre-OST0001 complete [ 4241.389054] LustreError: 133777:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4241.412422] LustreError: 133777:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4243.937041] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4243.946539] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4243.963113] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4246.500715] LustreError: 132073:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4246.523635] LustreError: 132073:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4248.486447] Lustre: server umount lustre-OST0000 complete [ 4255.200948] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4255.211388] LustreError: Skipped 1 previous similar message [ 4255.214328] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4255.222624] Lustre: Skipped 1 previous similar message [ 4255.226598] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4255.229437] Lustre: Skipped 3 previous similar messages [ 4259.043464] Lustre: server umount lustre-MDT0000 complete [ 4261.857620] LustreError: 131187:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4261.878203] LustreError: 131187:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4263.672442] LustreError: 130456:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781001098 with bad export cookie 227212006959539692 [ 4263.673382] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4263.685218] LustreError: 130456:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4263.963874] Lustre: server umount lustre-MDT0001 complete [ 4284.662372] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 4287.336282] Key type lgssc unregistered [ 4287.625491] LNet: 136313:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4287.634688] LNetError: 136313:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4287.647574] LNet: Removed LNI 192.168.203.154@tcp [ 4288.535984] Key type .llcrypt unregistered [ 4288.539038] Key type ._llcrypt unregistered [ 4331.795927] Key type ._llcrypt registered [ 4331.798978] Key type .llcrypt registered [ 4331.902686] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_hostid [ 4349.263381] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 4350.812029] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4350.836747] alg: No test for adler32 (adler32-zlib) [ 4352.087823] Lustre: Lustre: Build Version: 2.17.54_1_g695b805 [ 4352.386536] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 4354.088183] Key type lgssc registered [ 4355.529277] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4393.828067] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4393.881390] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4395.181369] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4395.236494] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4395.356376] Lustre: lustre-MDT0000: new disk, initializing [ 4395.489870] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4395.517970] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4400.350165] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 4412.111993] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4412.244896] Lustre: 140440:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4412.287497] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4412.291473] Lustre: Skipped 1 previous similar message [ 4412.352891] Lustre: lustre-MDT0001: new disk, initializing [ 4412.465781] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4412.513985] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4412.528853] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4417.387714] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 4426.604945] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4431.129254] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4438.536889] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4438.890508] Lustre: lustre-OST0000: new disk, initializing [ 4438.895613] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4438.908910] Lustre: 142180:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4439.061868] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4445.644677] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 4446.274172] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4446.285662] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4446.371110] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4453.472732] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4457.680120] Lustre: Failing over lustre-OST0000 [ 4459.864983] Lustre: server umount lustre-OST0000 complete [ 4475.872236] Lustre: 138662:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781001294/real 1781001294] req@00000000bd03301a x1867515100693376/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1781001310 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4475.922410] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4477.961143] Lustre: 138664:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781001296/real 1781001296] req@00000000558afd9c x1867515100693504/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1781001312 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4477.997456] Lustre: 138664:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4478.005653] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4478.836393] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4479.285401] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4479.304185] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4483.339281] Lustre: 138663:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781001302/real 1781001302] req@00000000095aefbd x1867515100695424/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1781001318 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4484.941524] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 4485.425463] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4487.552899] Lustre: lustre-OST0000: Recovery over after 0:02, of 3 clients 3 recovered and 0 were evicted. [ 4487.556869] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4488.619389] Lustre: 138663:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781001307/real 1781001307] req@000000008d5299a9 x1867515100696448/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1781001323 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4488.662804] Lustre: 138663:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4489.188950] Lustre: Failing over lustre-MDT0000 [ 4489.208347] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4489.234814] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4489.264585] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4489.584299] Lustre: server umount lustre-MDT0000 complete [ 4495.848608] LustreError: 140447:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4495.868240] LustreError: 140447:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 4497.427079] LustreError: 143323:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4501.002826] LustreError: 140451:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4501.029022] LustreError: 140451:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4506.086822] LustreError: 140450:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4506.124890] LustreError: 140450:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4507.106259] Lustre: 138665:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781001325/real 1781001325] req@000000000c803175 x1867515100709120/t0(0) o400->MGC192.168.203.154@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1781001341 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4507.154942] Lustre: 138665:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4507.160640] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4510.431257] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4512.228431] LustreError: 141462:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4512.253136] LustreError: 141462:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 4517.353911] LustreError: 138661:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000250ad50d x1867515100715136/t0(0) o250->MGC192.168.203.154@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4517.668378] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4517.743807] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4517.903116] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4522.385750] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 4522.991815] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4523.000534] Lustre: Skipped 1 previous similar message [ 4523.024787] Lustre: lustre-MDT0000: Recovery over after 0:06, of 2 clients 2 recovered and 0 were evicted. [ 4523.072815] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4533.218728] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4533.240282] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4533.249843] Lustre: Skipped 3 previous similar messages [ 4533.268205] Lustre: Skipped 1 previous similar message [ 4538.338423] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4538.346048] Lustre: Skipped 1 previous similar message [ 4539.064359] Lustre: server umount lustre-OST0000 complete [ 4548.579856] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4548.581222] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4548.618619] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4548.633177] Lustre: Skipped 1 previous similar message [ 4553.697617] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4553.710143] Lustre: Skipped 1 previous similar message [ 4558.305324] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4558.419546] Lustre: server umount lustre-MDT0000 complete [ 4558.817376] LustreError: 140446:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4558.829091] LustreError: 140446:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 4562.890597] LustreError: 140431:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781001397 with bad export cookie 17528280862434004662 [ 4562.898451] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4562.903788] LustreError: 140431:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4563.166438] Lustre: server umount lustre-MDT0001 complete [ 4577.925514] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 4581.486798] Key type lgssc unregistered [ 4581.830786] LNet: 145678:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4581.847402] LNetError: 145678:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4581.890794] LNet: Removed LNI 192.168.203.154@tcp [ 4583.021410] Key type .llcrypt unregistered [ 4583.025088] Key type ._llcrypt unregistered [ 4623.242796] Key type ._llcrypt registered [ 4623.245126] Key type .llcrypt registered [ 4623.358442] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 4624.508373] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4624.658730] alg: No test for adler32 (adler32-zlib) [ 4625.913715] Lustre: Lustre: Build Version: 2.17.54_1_g695b805 [ 4626.170878] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 4627.904309] Key type lgssc registered [ 4629.261593] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4637.992237] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4638.029772] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4639.643473] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4644.307218] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 4649.954946] LustreError: 147183:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4649.969663] LustreError: 147183:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4651.558306] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4655.074421] LustreError: 147182:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4655.367610] LustreError: 147211:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4655.384328] LustreError: 147211:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4655.405853] LustreError: 147211:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 16, retries 0, failed: rc = -5 [ 4655.926732] Lustre: server umount lustre-MDT0000 complete [ 4670.930771] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_hostid [ 4680.406331] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 4718.935615] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4719.158676] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4719.184570] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4719.242412] Lustre: lustre-MDT0000: new disk, initializing [ 4719.299704] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4719.312826] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4724.012369] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 4737.761136] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4737.922641] Lustre: 150372:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4738.039526] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4738.043800] Lustre: Skipped 1 previous similar message [ 4738.128459] Lustre: lustre-MDT0001: new disk, initializing [ 4738.328581] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4738.396736] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4738.410715] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4744.573173] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 4752.192968] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4756.304858] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4763.779977] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4764.079735] Lustre: lustre-OST0000: new disk, initializing [ 4764.088429] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4764.094630] Lustre: 152116:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4764.222542] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4770.664861] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 4772.414975] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4772.424671] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4772.596961] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4779.653384] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4784.761385] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4786.650214] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 06:40:19 (1781001619) [ 5080.038984] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5080.043032] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5080.082586] Lustre: Skipped 2 previous similar messages [ 5080.088740] Lustre: Skipped 1 previous similar message [ 5085.155215] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5085.166655] Lustre: Skipped 3 previous similar messages [ 5090.276826] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5090.290233] Lustre: Skipped 2 previous similar messages [ 5091.298507] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5091.543655] Lustre: server umount lustre-MDT0000 complete [ 5095.411373] LustreError: 150381:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5095.436850] LustreError: 150381:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5095.454310] LustreError: 150363:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781001930 with bad export cookie 6078114197110988835 [ 5095.463056] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5100.515642] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5100.517166] LustreError: 153767:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5100.557198] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5101.970337] Lustre: server umount lustre-MDT0001 complete [ 5112.248296] Lustre: server umount lustre-OST0000 complete [ 5122.940117] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_hostid [ 5138.347294] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 5184.888109] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5185.174821] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5185.221192] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5185.379206] Lustre: lustre-MDT0000: new disk, initializing [ 5185.568950] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5185.615798] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5190.221710] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5202.452986] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5202.582822] Lustre: 156640:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5202.653120] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5202.659350] Lustre: Skipped 1 previous similar message [ 5202.750182] Lustre: lustre-MDT0001: new disk, initializing [ 5202.849422] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5202.899106] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5202.915117] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5207.002345] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5215.091974] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5219.034317] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5226.125858] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5226.413322] Lustre: lustre-OST0000: new disk, initializing [ 5226.417827] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5226.424666] Lustre: 158382:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5226.515129] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5232.673354] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5233.221720] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5233.234723] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5233.396988] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5241.021718] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5246.343715] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5246.549377] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5251.712664] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5252.003249] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5254.113127] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5254.114783] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5254.123024] LustreError: Skipped 1 previous similar message [ 5254.123712] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5254.156748] Lustre: Skipped 1 previous similar message [ 5260.345522] Lustre: server umount lustre-OST0000 complete [ 5264.356316] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5264.363756] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5264.381821] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5264.394142] Lustre: Skipped 4 previous similar messages [ 5270.551535] Lustre: server umount lustre-MDT0000 complete [ 5274.601121] LustreError: 156648:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5274.623886] LustreError: 156648:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5274.746329] LustreError: 156632:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781002109 with bad export cookie 6078114197111147665 [ 5274.755925] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5274.762674] LustreError: 156632:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 5275.044069] Lustre: server umount lustre-MDT0001 complete [ 5284.926821] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 06:48:37 (1781002117) [ 5302.178374] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_hostid [ 5314.611611] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 5355.942625] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5356.261634] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5356.326539] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5356.465143] Lustre: lustre-MDT0000: new disk, initializing [ 5356.579329] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5356.604289] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5361.515695] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5372.806412] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5372.989622] Lustre: 162956:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5373.407946] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5378.478553] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5387.895780] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5393.182409] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5401.494384] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5401.879406] Lustre: lustre-OST0000: new disk, initializing [ 5401.886182] Lustre: Skipped 1 previous similar message [ 5401.899168] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5401.906710] Lustre: Skipped 2 previous similar messages [ 5401.918717] Lustre: 164707:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5409.480565] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5410.948599] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5410.964264] Lustre: Skipped 1 previous similar message [ 5410.976033] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5411.114971] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5417.752146] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5426.661620] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5426.662335] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5426.669990] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5426.669998] Lustre: Skipped 2 previous similar messages [ 5426.684127] LustreError: Skipped 1 previous similar message [ 5426.707449] Lustre: Skipped 2 previous similar messages [ 5432.692217] Lustre: server umount lustre-OST0000 complete [ 5439.969482] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5439.979442] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5443.576944] Lustre: server umount lustre-MDT0000 complete [ 5447.140990] LustreError: 162966:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5447.184972] LustreError: 162966:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5447.998309] LustreError: 163738:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781002282 with bad export cookie 6078114197111148547 [ 5448.004323] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5448.011729] LustreError: 163738:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5448.380502] Lustre: server umount lustre-MDT0001 complete [ 5460.430389] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 06:51:33 (1781002293) [ 5479.747070] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_hostid [ 5491.815425] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 5535.513269] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5535.776623] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5535.824659] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5535.960581] Lustre: lustre-MDT0000: new disk, initializing [ 5536.057697] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5536.064759] Lustre: Skipped 2 previous similar messages [ 5536.083416] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5540.984883] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5552.232299] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5552.433219] Lustre: 168890:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5552.775106] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5557.680455] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5566.251881] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5570.605928] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5578.361387] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5578.664195] Lustre: 170641:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5585.639712] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5587.522056] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5587.644808] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5593.929543] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5603.302198] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5603.306903] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5603.312139] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5603.331805] Lustre: Skipped 2 previous similar messages [ 5603.334824] Lustre: Skipped 6 previous similar messages [ 5608.437556] Lustre: server umount lustre-OST0000 complete [ 5618.946097] Lustre: server umount lustre-MDT0000 complete [ 5623.779767] LustreError: 168901:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5623.816827] LustreError: 168901:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5624.022844] LustreError: 168880:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781002458 with bad export cookie 6078114197111149632 [ 5624.045966] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5624.049491] LustreError: 168880:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5624.528768] Lustre: server umount lustre-MDT0001 complete [ 5636.582146] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 5640.191641] Key type lgssc unregistered [ 5640.551992] LNet: 172367:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5640.563169] LNetError: 172367:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5640.582886] LNet: Removed LNI 192.168.203.154@tcp [ 5641.495295] Key type .llcrypt unregistered [ 5641.501434] Key type ._llcrypt unregistered [ 5668.452188] Key type ._llcrypt registered [ 5668.453963] Key type .llcrypt registered [ 5668.560721] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 5669.856748] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5669.872413] alg: No test for adler32 (adler32-zlib) [ 5671.144328] Lustre: Lustre: Build Version: 2.17.54_1_g695b805 [ 5671.500873] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 5673.250935] Key type lgssc registered [ 5674.400833] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5683.678799] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5683.701646] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5685.207288] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5689.655573] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5695.467187] LustreError: 173774:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5695.496334] LustreError: 173774:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5699.133870] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5699.610154] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5704.367044] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5710.998158] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5715.044876] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5722.358861] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5722.808209] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5729.390954] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5731.087078] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5736.289652] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5755.878077] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5755.887897] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5755.911628] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5756.898816] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5756.903431] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5756.943407] Lustre: Skipped 1 previous similar message [ 5761.529955] Lustre: server umount lustre-OST0000 complete [ 5766.115511] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5766.122842] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5766.143376] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5772.207602] Lustre: server umount lustre-MDT0000 complete [ 5772.259866] LustreError: 173769:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5772.288286] LustreError: 173769:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5776.350390] LustreError: 173755:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781002611 with bad export cookie 8320967234655208270 [ 5776.353763] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5776.371830] LustreError: 173755:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5776.775460] Lustre: server umount lustre-MDT0001 complete [ 5789.138191] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 5792.409127] Key type lgssc unregistered [ 5792.841071] LNet: 176973:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5792.855136] LNetError: 176973:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5792.879399] LNet: Removed LNI 192.168.203.154@tcp [ 5793.960657] Key type .llcrypt unregistered [ 5793.962247] Key type ._llcrypt unregistered [ 5819.794528] Key type ._llcrypt registered [ 5819.796353] Key type .llcrypt registered [ 5819.919244] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 5821.315365] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5821.366417] alg: No test for adler32 (adler32-zlib) [ 5822.407837] Lustre: Lustre: Build Version: 2.17.54_1_g695b805 [ 5822.651675] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 5824.371044] Key type lgssc registered [ 5825.864707] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5834.324446] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5834.375954] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5835.986114] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5840.830813] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5845.512737] LustreError: 178374:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5845.538302] LustreError: 178374:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5850.611135] LustreError: 178375:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5851.470452] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5851.916866] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5857.549624] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5864.418804] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5869.939166] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5879.156982] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5879.789167] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5880.832908] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5888.817454] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5896.628893] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5904.959156] Lustre: lustre-OST0000: Not available for connect from 192.168.203.54@tcp (stopping) [ 5906.412280] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5906.415042] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5906.429187] LustreError: Skipped 1 previous similar message [ 5906.440193] Lustre: Skipped 1 previous similar message [ 5906.445474] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5906.450498] Lustre: Skipped 1 previous similar message [ 5910.030069] Lustre: lustre-OST0000: Not available for connect from 192.168.203.54@tcp (stopping) [ 5910.432747] Lustre: server umount lustre-OST0000 complete [ 5917.534420] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff9fc95a3bb800.ost_server_uuid 50 [ 5919.516880] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5933.596523] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5933.627344] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5933.642885] Lustre: Skipped 1 previous similar message [ 5938.658803] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5938.671489] Lustre: Skipped 1 previous similar message [ 5939.884515] Lustre: server umount lustre-MDT0000 complete [ 5943.780045] LustreError: 178375:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5943.804198] LustreError: 178375:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5943.916183] LustreError: 178356:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781002778 with bad export cookie 3293974510973894489 [ 5943.941949] LustreError: 178356:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5943.950672] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5944.311934] Lustre: server umount lustre-MDT0001 complete [ 5954.754929] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 06:59:47 (1781002787) [ 5962.551455] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5963.059935] LustreError: 181854:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5963.205732] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5968.132065] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5968.357235] LustreError: 181855:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5977.362381] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5977.793860] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5983.560946] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5989.382437] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5993.612948] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6002.149086] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6002.696060] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6008.849717] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6009.229310] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 6015.628051] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6024.140804] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6024.356930] Lustre: lustre-OST0001: new disk, initializing [ 6024.360088] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6024.364830] Lustre: 184356:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6024.457990] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6032.428834] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 6032.990204] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6033.012875] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6033.122094] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6041.406830] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6044.643249] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6044.657934] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6044.667662] Lustre: Skipped 1 previous similar message [ 6044.671831] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6048.753304] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6048.759719] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6050.955355] Lustre: server umount lustre-OST0000 complete [ 6052.872253] LustreError: 183451:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6052.899266] LustreError: 183451:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6059.895658] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6060.271864] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6065.542678] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6065.890734] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6069.220205] LustreError: 183451:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6069.242095] LustreError: 183451:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 10 previous similar messages [ 6084.578397] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6084.584474] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6084.592590] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6084.614544] Lustre: Skipped 1 previous similar message [ 6084.635211] Lustre: Skipped 4 previous similar messages [ 6087.178565] Lustre: server umount lustre-OST0001 complete [ 6094.830626] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6094.856537] Lustre: Skipped 1 previous similar message [ 6098.249913] Lustre: server umount lustre-MDT0000 complete [ 6102.540227] LustreError: 184367:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781002937 with bad export cookie 3293974510973895560 [ 6102.550590] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6102.551098] LustreError: 184367:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6103.184654] Lustre: server umount lustre-MDT0001 complete [ 6131.884331] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 07:02:44 (1781002964) [ 6139.552650] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6139.872684] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6139.981539] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6140.265412] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6145.412951] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 6155.368467] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6155.545323] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6155.632222] Lustre: 187660:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6161.416224] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 6168.191490] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6173.343246] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6181.595517] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6181.836544] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6188.679271] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6188.913863] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 6195.172290] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6203.193157] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6203.282489] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6210.719598] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 6217.678712] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6223.902350] Lustre: lustre-OST0000: Not available for connect from 192.168.203.54@tcp (stopping) [ 6223.919852] Lustre: Skipped 2 previous similar messages [ 6224.356251] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6227.060355] Lustre: server umount lustre-OST0000 complete [ 6229.030753] LustreError: 189286:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6229.068645] LustreError: 189286:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 6244.840848] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6244.854788] LustreError: Skipped 1 previous similar message [ 6250.257600] Lustre: server umount lustre-OST0001 complete [ 6260.816137] Lustre: server umount lustre-MDT0000 complete [ 6265.175274] LustreError: 190211:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781003100 with bad export cookie 3293974510973896967 [ 6265.178528] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6265.183125] LustreError: 190211:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6265.842783] Lustre: server umount lustre-MDT0001 complete [ 6290.490000] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 07:05:23 (1781003123) [ 6306.468867] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_hostid [ 6315.180127] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 6356.155641] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6356.512478] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6356.572087] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6356.675497] Lustre: lustre-MDT0000: new disk, initializing [ 6356.829775] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6356.837813] Lustre: Skipped 3 previous similar messages [ 6356.857692] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6361.224275] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 6372.935534] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6373.062582] Lustre: 195273:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6373.118091] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6373.123095] Lustre: Skipped 1 previous similar message [ 6373.248070] Lustre: lustre-MDT0001: new disk, initializing [ 6373.503709] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6373.524288] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6377.798156] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 6386.251893] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6390.384372] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6397.980994] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6398.441496] Lustre: lustre-OST0000: new disk, initializing [ 6398.447640] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6398.458131] Lustre: 197017:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6404.892682] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 6406.780802] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6406.791943] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6406.995559] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6414.231526] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6421.595768] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6421.907796] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6426.702470] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6426.938908] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6432.750748] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6432.757463] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6432.764224] LustreError: Skipped 1 previous similar message [ 6432.768658] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6432.798465] Lustre: Skipped 6 previous similar messages [ 6432.812466] Lustre: Skipped 10 previous similar messages [ 6434.798536] Lustre: server umount lustre-OST0000 complete [ 6445.236408] Lustre: server umount lustre-MDT0000 complete [ 6448.099202] LustreError: 196612:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6448.122474] LustreError: 196612:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 13 previous similar messages [ 6449.010111] LustreError: 195266:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781003283 with bad export cookie 3293974510973898521 [ 6449.015059] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6449.021042] LustreError: 195266:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6449.480750] Lustre: server umount lustre-MDT0001 complete [ 6457.353861] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6462.382440] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 6471.391587] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6475.961408] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 6482.062378] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6486.433963] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6492.709121] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6493.042084] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6493.046309] Lustre: Skipped 4 previous similar messages [ 6499.390085] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 6505.461662] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6509.846112] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6510.070349] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6514.315204] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6514.567700] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6516.714838] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6516.738916] LustreError: Skipped 1 previous similar message [ 6516.741722] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6516.757145] Lustre: Skipped 2 previous similar messages [ 6522.876251] Lustre: server umount lustre-OST0000 complete [ 6529.325752] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6534.223345] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6562.561302] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6567.970614] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6567.982911] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6568.210522] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 6573.375467] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6581.696409] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 6 sec [ 6582.314024] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6582.335537] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6587.370150] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6587.376039] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6587.380095] LustreError: Skipped 1 previous similar message [ 6587.394582] Lustre: Skipped 7 previous similar messages [ 6592.374345] Lustre: server umount lustre-OST0000 complete [ 6601.825500] Lustre: server umount lustre-MDT0000 complete [ 6602.723062] LustreError: 203325:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6602.747259] LustreError: 203325:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 6605.274213] LustreError: 199072:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781003440 with bad export cookie 3293974510973899410 [ 6605.276058] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6605.289716] LustreError: 199072:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6605.591191] Lustre: server umount lustre-MDT0001 complete [ 6613.380537] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 07:10:46 (1781003446) [ 6618.581812] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6622.244352] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 6630.664209] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6635.299193] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 6641.816697] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6646.302210] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6653.948253] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6661.310295] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 6667.258902] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6672.072871] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6672.297892] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6676.590980] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6676.798242] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6684.049822] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6684.285082] Lustre: lustre-OST0001: new disk, initializing [ 6684.290043] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6684.299443] Lustre: 207532:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6690.359859] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 6692.434355] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6692.448130] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6692.531831] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6697.804516] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6701.889731] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6702.083732] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6706.464756] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6706.670632] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6713.321237] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6713.321461] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6713.339656] Lustre: Skipped 5 previous similar messages [ 6713.360951] LustreError: Skipped 1 previous similar message [ 6714.447856] Lustre: server umount lustre-OST0001 complete [ 6720.128727] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6720.296188] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6724.106348] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6724.367377] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6752.725111] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6752.970426] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6752.978627] Lustre: Skipped 5 previous similar messages [ 6758.382815] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6758.418401] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6758.420374] Lustre: lustre-OST0001-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6759.393396] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 6765.480667] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6767.322665] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6775.602191] Lustre: server umount lustre-OST0001 complete [ 6799.057607] Lustre: server umount lustre-OST0000 complete [ 6808.741307] Lustre: server umount lustre-MDT0000 complete [ 6812.229271] LustreError: 204515:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781003647 with bad export cookie 3293974510973900376 [ 6812.236946] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6812.239086] LustreError: 204515:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6818.632507] Lustre: server umount lustre-MDT0001 complete [ 6840.720127] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 07:14:33 (1781003673) [ 6847.831706] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6848.294338] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6848.353308] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6852.938494] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 6863.137179] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6863.357799] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6863.375544] Lustre: 212479:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6867.845041] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 6873.523680] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6877.277907] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6883.893164] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6884.134763] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6890.955459] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 6898.283654] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6910.942475] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6911.092270] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6918.771861] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 6925.724063] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6927.141777] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6932.224840] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6932.448794] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6937.285740] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6937.526965] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6941.164978] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9fc947fde000.ost_server_uuid 50 [ 6942.823966] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9fc947fde000.ost_server_uuid in IDLE state after 0 sec [ 6943.857457] Lustre: Permanently deactivating lustre-OST0001 [ 6943.891458] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6947.864851] Lustre: Permanently reactivating lustre-OST0001 [ 6947.901717] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 6947.912536] Lustre: Skipped 2 previous similar messages [ 6952.431319] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6952.433671] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6952.443745] Lustre: Skipped 12 previous similar messages [ 6952.448703] LustreError: Skipped 3 previous similar messages [ 6954.295772] Lustre: server umount lustre-OST0001 complete [ 6957.560354] LustreError: 215022:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6957.586572] LustreError: 215022:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 35 previous similar messages [ 6964.729497] Lustre: server umount lustre-OST0000 complete [ 6971.373766] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6971.400028] Lustre: Skipped 11 previous similar messages [ 6975.895561] Lustre: server umount lustre-MDT0000 complete [ 6980.436875] LustreError: 212472:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781003815 with bad export cookie 3293974510973901664 [ 6980.451672] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6980.462510] LustreError: 212472:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6981.022762] Lustre: server umount lustre-MDT0001 complete [ 7002.969706] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 07:17:16 (1781003836) [ 7010.709373] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7010.974157] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7011.008111] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7011.020631] Lustre: Skipped 2 previous similar messages [ 7015.064082] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 7024.154574] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7024.233640] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7024.255067] Lustre: 218591:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7029.205588] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 7035.147367] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7038.791767] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7045.076521] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7045.247429] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7045.294520] Lustre: Permanently deactivating lustre-OST0000 [ 7045.343917] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7050.785936] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7052.051771] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 7058.652768] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7066.137153] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7066.262442] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7072.927805] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7073.200283] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 7078.921873] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7080.786753] Lustre: Permanently reactivating lustre-OST0000 [ 7080.811227] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7080.819397] Lustre: Skipped 2 previous similar messages [ 7082.790765] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7082.790789] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 32 seconds. I think it's dead, and I am evicting it. exp 000000004ad3da8d, cur 1781003917 deadline 1781003915 last 1781003885 [ 7082.796783] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7082.831094] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7082.833858] Lustre: Skipped 1 previous similar message [ 7082.927060] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7096.249651] Lustre: server umount lustre-OST0001 complete [ 7106.235146] Lustre: server umount lustre-OST0000 complete [ 7116.367961] Lustre: server umount lustre-MDT0000 complete [ 7119.970956] LustreError: 218585:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781003954 with bad export cookie 3293974510973903337 [ 7119.977697] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7119.986475] LustreError: 218585:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7120.374150] Lustre: server umount lustre-MDT0001 complete [ 7131.988349] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 7134.656783] Key type lgssc unregistered [ 7134.970783] LNet: 223004:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7134.980863] LNetError: 223004:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7135.008456] LNet: Removed LNI 192.168.203.154@tcp [ 7135.865184] Key type .llcrypt unregistered [ 7135.868746] Key type ._llcrypt unregistered [ 7158.847947] Key type ._llcrypt registered [ 7158.851480] Key type .llcrypt registered [ 7158.985589] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 7160.661604] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7160.702545] alg: No test for adler32 (adler32-zlib) [ 7161.751818] Lustre: Lustre: Build Version: 2.17.54_1_g695b805 [ 7162.010087] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 7163.680186] Key type lgssc registered [ 7164.720163] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7173.970248] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7173.998745] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7175.527526] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7179.691769] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 7185.894367] LustreError: 224456:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7185.916301] LustreError: 224456:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7188.167442] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7188.287744] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7188.333492] Lustre: 224446:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7188.344893] Lustre: Permanently deactivating lustre-MDT0001 [ 7188.360655] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7188.707714] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7192.109358] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7192.113973] LustreError: 224487:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 7192.134794] LustreError: 224487:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 7192.152691] LustreError: 224487:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 17, retries 1, failed: rc = -108 [ 7192.698887] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 7198.222309] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7202.396308] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7211.291783] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7211.869694] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7213.953841] LustreError: 226069:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7214.004095] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 7218.436874] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 7219.173203] LustreError: 226071:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7220.221886] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7224.306996] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7225.313379] LustreError: 226188:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7225.344384] LustreError: 226188:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7230.453301] LustreError: 226070:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7230.483342] LustreError: 226070:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7231.180399] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7231.480682] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7236.593261] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7236.619196] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7237.810613] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 7243.219535] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7247.051739] Lustre: Permanently reactivating lustre-MDT0001 [ 7247.063777] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7247.067601] Lustre: Skipped 2 previous similar messages [ 7250.410625] Lustre: lustre-MDT0001: Client 7c1160ef-e03a-40b1-9d9c-1423593f8d14 (at 192.168.203.54@tcp) reconnecting [ 7253.801476] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7255.326522] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7260.402768] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7260.631317] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7264.961657] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7265.187784] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7267.658417] Lustre: Permanently deactivating lustre-MDT0001 [ 7267.668306] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7267.672774] Lustre: Skipped 2 previous similar messages [ 7275.022452] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7282.658412] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7282.664591] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7282.678593] Lustre: Skipped 3 previous similar messages [ 7282.696438] Lustre: Skipped 3 previous similar messages [ 7285.583759] Lustre: server umount lustre-MDT0000 complete [ 7287.784508] LustreError: 226915:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7287.805473] LustreError: 226915:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7288.718328] LustreError: 224437:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781004123 with bad export cookie 10133633825972632678 [ 7288.721686] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7288.727014] LustreError: 224437:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7288.946771] Lustre: server umount lustre-MDT0001 complete [ 7298.540959] Lustre: server umount lustre-OST0000 complete [ 7311.306426] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 07:22:24 (1781004144) [ 7323.300148] Lustre: server umount lustre-OST0001 complete [ 7330.930184] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_hostid [ 7342.189392] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 7380.160557] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7380.436845] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7380.447899] Lustre: Skipped 2 previous similar messages [ 7380.484110] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7380.574387] Lustre: lustre-MDT0000: new disk, initializing [ 7380.642636] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7380.661892] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7384.865776] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 7396.841956] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7396.988818] Lustre: 232097:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7397.036871] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7397.041971] Lustre: Skipped 1 previous similar message [ 7397.121262] Lustre: lustre-MDT0001: new disk, initializing [ 7397.220492] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7397.258525] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7397.273163] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7402.495869] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 7410.065702] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7414.312162] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7421.639368] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7422.048858] Lustre: lustre-OST0000: new disk, initializing [ 7422.054122] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7422.066944] Lustre: 233836:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7422.158656] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7428.154230] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7428.176557] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7428.277723] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7428.528641] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 7436.186859] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7438.359612] LustreError: 233843:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7446.232368] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7446.409432] Lustre: lustre-OST0001: new disk, initializing [ 7446.412838] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7446.418358] Lustre: 234840:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7448.432461] LustreError: 233843:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7448.473819] LustreError: 234682:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7452.204392] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 7453.755851] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7453.778304] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7453.944653] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7458.568177] LustreError: 234682:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7459.491638] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7461.189336] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7464.429883] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7464.435234] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7464.446367] LustreError: Skipped 1 previous similar message [ 7464.448830] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7464.482122] Lustre: Skipped 1 previous similar message [ 7467.075269] Lustre: lustre-OST0001: Not available for connect from 192.168.203.54@tcp (stopping) [ 7467.084788] Lustre: Skipped 1 previous similar message [ 7469.224452] Lustre: server umount lustre-OST0001 complete [ 7469.542105] LustreError: 235506:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7469.551917] LustreError: 235506:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7490.019713] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7490.027611] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7490.051754] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7490.057850] Lustre: Skipped 1 previous similar message [ 7491.354092] Lustre: server umount lustre-OST0000 complete [ 7495.138949] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7495.143701] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7495.156569] Lustre: Skipped 2 previous similar messages [ 7501.156971] Lustre: server umount lustre-MDT0000 complete [ 7504.337704] LustreError: 232087:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781004339 with bad export cookie 10133633825972635492 [ 7504.343292] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7504.351241] LustreError: 232087:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7510.740712] Lustre: server umount lustre-MDT0001 complete [ 7520.136120] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 7523.109237] Key type lgssc unregistered [ 7523.446476] LNet: 236746:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7523.456247] LNetError: 236746:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7523.481641] LNet: Removed LNI 192.168.203.154@tcp [ 7524.543645] Key type .llcrypt unregistered [ 7524.546855] Key type ._llcrypt unregistered [ 7565.717515] Key type ._llcrypt registered [ 7565.720614] Key type .llcrypt registered [ 7565.818457] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 7566.763811] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7566.812592] alg: No test for adler32 (adler32-zlib) [ 7567.880072] Lustre: Lustre: Build Version: 2.17.54_1_g695b805 [ 7568.227122] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 7569.944237] Key type lgssc registered [ 7571.277494] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7578.778052] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7578.809521] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7580.121318] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7580.141372] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7580.246324] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7583.889577] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 7592.882101] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7593.010167] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7593.064852] Lustre: 238935:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7593.338215] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7597.155766] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 7602.939813] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7606.257692] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7612.830408] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7612.959327] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7613.151611] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7618.822191] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 7621.777904] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 7623.324888] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7672.801712] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7672.808186] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7672.818155] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7673.313488] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7673.321629] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7673.331729] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7673.340140] Lustre: Skipped 1 previous similar message [ 7677.693940] Lustre: server umount lustre-OST0000 complete [ 7681.448977] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7692.894972] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7693.243107] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7694.257261] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7694.272974] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7698.410799] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7698.421856] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7698.965152] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 7703.698968] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7744.482281] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7744.488444] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7744.494724] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7744.511550] Lustre: Skipped 1 previous similar message [ 7744.522067] Lustre: Skipped 1 previous similar message [ 7748.500851] Lustre: server umount lustre-OST0000 complete [ 7754.724376] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7754.745163] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7758.621496] Lustre: server umount lustre-MDT0000 complete [ 7759.845112] LustreError: 238943:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7759.866935] LustreError: 238943:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7762.332527] LustreError: 239675:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781004597 with bad export cookie 12157833388635641545 [ 7762.337594] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7762.351850] LustreError: 239675:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7762.887154] Lustre: server umount lustre-MDT0001 complete [ 7774.391607] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 7777.370781] Key type lgssc unregistered [ 7777.697164] LNet: 243886:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7777.704643] LNetError: 243886:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7777.718643] LNet: Removed LNI 192.168.203.154@tcp [ 7778.483344] Key type .llcrypt unregistered [ 7778.486251] Key type ._llcrypt unregistered [ 7803.425303] Key type ._llcrypt registered [ 7803.430891] Key type .llcrypt registered [ 7803.587045] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 7805.088229] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7805.110216] alg: No test for adler32 (adler32-zlib) [ 7806.237303] Lustre: Lustre: Build Version: 2.17.54_1_g695b805 [ 7806.525507] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 7808.230121] Key type lgssc registered [ 7809.554822] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7817.599507] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7817.619277] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7819.149809] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7823.173783] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 7829.473976] LustreError: 245288:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7829.493391] LustreError: 245288:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7830.815737] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7831.199260] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7835.328259] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 7840.630907] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7843.986887] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7850.001979] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7850.384298] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7856.210590] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 7862.198115] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7864.775039] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 7890.402337] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7890.406407] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7890.406698] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7890.422261] Lustre: Skipped 1 previous similar message [ 7890.448235] Lustre: Skipped 1 previous similar message [ 7895.549454] Lustre: server umount lustre-OST0000 complete [ 7900.643638] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7900.645399] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7900.668635] Lustre: Skipped 1 previous similar message [ 7905.297246] Lustre: server umount lustre-MDT0000 complete [ 7905.770672] LustreError: 245288:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7905.789918] LustreError: 245288:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7909.190094] LustreError: 245268:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781004744 with bad export cookie 1975235812725026304 [ 7909.195576] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7909.195711] LustreError: 245268:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7909.474314] Lustre: server umount lustre-MDT0001 complete [ 7918.492580] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 7921.655288] Key type lgssc unregistered [ 7921.940451] LNet: 249237:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7921.953817] LNetError: 249237:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7921.990417] LNet: Removed LNI 192.168.203.154@tcp [ 7922.933608] Key type .llcrypt unregistered [ 7922.939453] Key type ._llcrypt unregistered [ 7939.628691] Key type ._llcrypt registered [ 7939.632685] Key type .llcrypt registered [ 7939.842128] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 7940.561678] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7940.598678] alg: No test for adler32 (adler32-zlib) [ 7941.811438] Lustre: Lustre: Build Version: 2.17.54_1_g695b805 [ 7942.249423] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 7944.088287] Key type lgssc registered [ 7945.718510] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7955.261818] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7955.273519] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7956.695688] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7961.139703] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 7966.183285] LustreError: 250443:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7966.209112] LustreError: 250443:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7969.439227] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7969.870658] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7974.260557] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 7980.060341] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7983.726285] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7990.270284] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7990.648653] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7996.781788] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 8001.904496] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8002.703312] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8015.333977] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8015.339840] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8015.351665] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8015.603736] Lustre: server umount lustre-OST0000 complete [ 8020.961416] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8020.974944] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8020.995710] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8022.504802] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8022.507451] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8027.619652] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8027.623190] Lustre: Skipped 2 previous similar messages [ 8032.748425] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8032.755221] Lustre: Skipped 1 previous similar message [ 8034.273880] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8034.413472] Lustre: server umount lustre-MDT0000 complete [ 8037.857794] LustreError: 250437:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8037.867385] LustreError: 250437:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8038.397886] LustreError: 250422:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781004873 with bad export cookie 459032083617091299 [ 8038.404889] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8038.410370] LustreError: 250422:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8038.707183] Lustre: server umount lustre-MDT0001 complete [ 8049.417213] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 8052.239735] Key type lgssc unregistered [ 8052.567695] LNet: 253764:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8052.591320] LNetError: 253764:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8052.622447] LNet: Removed LNI 192.168.203.154@tcp [ 8053.291620] Key type .llcrypt unregistered [ 8053.293675] Key type ._llcrypt unregistered [ 8079.197989] Key type ._llcrypt registered [ 8079.201687] Key type .llcrypt registered [ 8079.283814] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 8080.498349] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8080.546092] alg: No test for adler32 (adler32-zlib) [ 8081.650815] Lustre: Lustre: Build Version: 2.17.54_1_g695b805 [ 8081.821702] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 8083.496450] Key type lgssc registered [ 8084.610879] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8092.607681] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8092.648898] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8094.125176] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8098.485493] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 8103.411520] LustreError: 255163:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8103.448907] LustreError: 255163:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8107.395032] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8107.791830] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8112.202263] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 8117.928813] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8121.848360] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8128.906743] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8129.336795] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8135.877369] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 8138.550154] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8142.166460] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8168.425344] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8168.430102] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8168.449633] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8169.445921] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8169.453514] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8169.467569] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8172.146364] Lustre: server umount lustre-OST0000 complete [ 8179.683502] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8179.686611] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8179.693324] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8179.693332] Lustre: Skipped 1 previous similar message [ 8181.911795] Lustre: server umount lustre-MDT0000 complete [ 8184.807119] LustreError: 255158:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8184.834430] LustreError: 255158:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8185.831311] LustreError: 255144:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781005020 with bad export cookie 12916145544452225614 [ 8185.846819] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8185.847225] LustreError: 255144:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8186.279828] Lustre: server umount lustre-MDT0001 complete [ 8197.210217] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 8200.018348] Key type lgssc unregistered [ 8200.416724] LNet: 259114:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8200.436698] LNetError: 259114:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8200.462775] LNet: Removed LNI 192.168.203.154@tcp [ 8201.388223] Key type .llcrypt unregistered [ 8201.395639] Key type ._llcrypt unregistered [ 8218.262426] Key type ._llcrypt registered [ 8218.266772] Key type .llcrypt registered [ 8218.411163] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 8219.256568] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8219.324223] alg: No test for adler32 (adler32-zlib) [ 8220.322808] Lustre: Lustre: Build Version: 2.17.54_1_g695b805 [ 8220.548429] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 8222.224355] Key type lgssc registered [ 8223.343818] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8230.621484] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8230.647023] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8232.083255] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8235.902862] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 8241.639351] LustreError: 260347:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8241.664117] LustreError: 260347:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8243.506670] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8243.805514] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8247.545933] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 8252.327942] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8255.708432] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8261.227030] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8261.690977] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8267.313663] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 8271.972515] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8277.045739] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8287.203372] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8287.215747] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8287.219676] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8287.231789] Lustre: Skipped 1 previous similar message [ 8287.242228] Lustre: Skipped 1 previous similar message [ 8291.544198] Lustre: server umount lustre-OST0000 complete [ 8294.881476] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8294.891547] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8294.905560] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8297.441020] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8297.442041] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8300.920743] Lustre: server umount lustre-MDT0000 complete [ 8302.564967] LustreError: 260346:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8302.581419] LustreError: 260346:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8303.993424] LustreError: 260311:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781005138 with bad export cookie 16300128271504586824 [ 8304.001129] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8304.016796] LustreError: 260311:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8304.424233] Lustre: server umount lustre-MDT0001 complete [ 8312.533575] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 8314.388257] Key type lgssc unregistered [ 8314.623498] LNet: 263689:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8314.636793] LNetError: 263689:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8315.687701] LNet: Removed LNI 192.168.203.154@tcp [ 8316.275812] Key type .llcrypt unregistered [ 8316.279157] Key type ._llcrypt unregistered [ 8324.564931] Key type ._llcrypt registered [ 8324.566224] Key type .llcrypt registered [ 8324.663172] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8339.388332] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_hostid [ 8351.559963] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 8352.445896] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8352.564763] alg: No test for adler32 (adler32-zlib) [ 8353.602499] Lustre: Lustre: Build Version: 2.17.54_1_g695b805 [ 8353.813108] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 8355.496223] Key type lgssc registered [ 8356.561317] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8386.688453] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8386.708897] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8387.959897] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8387.991637] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8388.062956] Lustre: lustre-MDT0000: new disk, initializing [ 8388.169807] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8388.201910] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8392.202556] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 8400.861165] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8400.964638] Lustre: 267167:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8401.025331] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8401.030214] Lustre: Skipped 1 previous similar message [ 8401.086253] Lustre: lustre-MDT0001: new disk, initializing [ 8401.170478] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8401.200440] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8401.207858] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8404.670827] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 8410.484986] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8413.435838] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8418.886936] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8419.154823] Lustre: lustre-OST0000: new disk, initializing [ 8419.165544] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8419.182734] Lustre: 268907:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8419.278085] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8424.262144] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 8427.560572] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8427.565975] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8427.632200] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8429.903036] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8433.634510] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8433.813656] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8437.420701] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8437.616249] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8442.850914] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8442.855790] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8442.856257] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8442.865466] Lustre: Skipped 1 previous similar message [ 8445.396330] Lustre: server umount lustre-OST0000 complete [ 8452.577502] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8452.584297] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8452.608984] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8452.611953] Lustre: Skipped 1 previous similar message [ 8454.960780] Lustre: server umount lustre-MDT0000 complete [ 8458.819301] LustreError: 267159:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781005293 with bad export cookie 16879520003926940082 [ 8458.822446] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8458.825996] LustreError: 267159:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8459.080523] Lustre: server umount lustre-MDT0001 complete [ 8467.662877] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 07:41:40 (1781005300) [ 8475.177024] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8475.693639] LustreError: 271172:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8475.707572] LustreError: 271172:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 8475.769745] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8480.536490] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 8481.248975] LustreError: 271173:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8485.346022] LustreError: 271172:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8488.653177] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8489.023669] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8492.968251] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 8498.559873] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8501.351704] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8507.147194] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8507.383403] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8512.602885] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 8517.386250] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8533.988340] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8534.002000] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8534.014687] Lustre: Skipped 1 previous similar message [ 8534.018478] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8534.022181] Lustre: Skipped 2 previous similar messages [ 8536.036623] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8536.038677] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8536.038799] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8536.066471] Lustre: Skipped 1 previous similar message [ 8537.444990] Lustre: server umount lustre-OST0000 complete [ 8541.156387] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8541.166218] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8541.170478] Lustre: Skipped 1 previous similar message [ 8547.204966] Lustre: server umount lustre-MDT0000 complete [ 8550.972499] LustreError: 271152:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781005385 with bad export cookie 16879520003926940964 [ 8550.973097] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8550.982269] LustreError: 271152:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8551.304195] Lustre: server umount lustre-MDT0001 complete [ 8559.817532] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 8562.065766] Key type lgssc unregistered [ 8562.322374] LNet: 274371:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8562.343986] LNetError: 274371:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8562.369740] LNet: Removed LNI 192.168.203.154@tcp [ 8563.048935] Key type .llcrypt unregistered [ 8563.051606] Key type ._llcrypt unregistered [ 8585.973959] Key type ._llcrypt registered [ 8585.976410] Key type .llcrypt registered [ 8586.076523] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_hostid [ 8600.757412] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 8601.662539] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8601.689097] alg: No test for adler32 (adler32-zlib) [ 8602.673827] Lustre: Lustre: Build Version: 2.17.54_1_g695b805 [ 8602.876869] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 8604.536161] Key type lgssc registered [ 8605.837918] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8653.435149] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8653.497884] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8654.733696] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8654.799270] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8654.880499] Lustre: lustre-MDT0000: new disk, initializing [ 8654.966369] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8654.990900] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8659.253505] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 8667.544332] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8667.657963] Lustre: 278166:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8667.759687] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8667.765021] Lustre: Skipped 1 previous similar message [ 8667.885698] Lustre: lustre-MDT0001: new disk, initializing [ 8667.960021] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8667.993378] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8668.009082] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8671.309647] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 8679.361296] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8682.799775] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8688.835570] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8689.187594] Lustre: lustre-OST2710: new disk, initializing [ 8689.191964] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 8689.204334] Lustre: 279906:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8689.287806] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 8694.713481] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 8699.443506] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 8699.453587] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 8699.573814] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 8702.514665] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 8708.321025] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8708.494627] Lustre: lustre-OST03e8: new disk, initializing [ 8708.498713] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 8708.504506] Lustre: 280857:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8708.590143] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 8712.979559] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 8715.844089] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 8715.855660] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 8715.925411] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 8717.462800] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 8722.302451] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 8722.476196] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8725.734576] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 8725.959812] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8730.016632] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 8730.198993] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8734.151769] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 8734.316334] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8739.814541] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8739.836242] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8739.864683] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8740.914590] Lustre: server umount lustre-MDT0000 complete [ 8741.873681] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8744.491126] LustreError: 278155:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781005579 with bad export cookie 14863485809946660321 [ 8744.501348] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8744.516033] LustreError: 278155:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8744.933857] Lustre: server umount lustre-MDT0001 complete [ 8759.197634] Lustre: server umount lustre-OST2710 complete [ 8762.337103] Lustre: 275963:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781005581/real 1781005581] req@000000003d6e4e46 x1867519557167616/t0(0) o400->lustre-MDT0001-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781005597 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8762.375403] Lustre: lustre-MDT0001-lwp-OST03e8: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8762.392601] Lustre: Skipped 2 previous similar messages [ 8764.022488] Lustre: server umount lustre-OST03e8 complete [ 8783.697756] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_hostid [ 8793.758800] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 8826.111641] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8826.376838] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8826.401913] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8826.518978] Lustre: lustre-MDT0000: new disk, initializing [ 8826.588054] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8826.608861] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8830.514335] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 8839.622360] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8839.713275] Lustre: 285885:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8839.774647] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8839.780473] Lustre: Skipped 1 previous similar message [ 8839.867219] Lustre: lustre-MDT0001: new disk, initializing [ 8839.953292] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8840.018063] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8840.031731] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8843.625571] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 8850.644127] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8853.849810] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8860.227860] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8860.454557] Lustre: lustre-OST0000: new disk, initializing [ 8860.457626] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8860.463531] Lustre: 287629:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8860.534772] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8866.517067] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 8867.906497] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8867.913760] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8868.034243] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8875.909498] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8880.746236] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8881.047538] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8886.658401] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8886.895338] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8893.925106] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8893.926524] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8893.929353] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8893.956287] Lustre: Skipped 1 previous similar message [ 8899.042182] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8899.048736] Lustre: Skipped 2 previous similar messages [ 8903.136232] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8903.262652] Lustre: server umount lustre-OST0000 complete [ 8909.286485] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8909.299839] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8909.301043] Lustre: Skipped 1 previous similar message [ 8909.318904] Lustre: Skipped 1 previous similar message [ 8913.503546] Lustre: server umount lustre-MDT0000 complete [ 8914.402387] LustreError: 287637:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8914.426390] LustreError: 287637:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 8917.996811] LustreError: 285880:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781005752 with bad export cookie 14863485809946662001 [ 8918.009986] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8918.015438] LustreError: 285880:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8918.453659] Lustre: server umount lustre-MDT0001 complete [ 8928.001994] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 07:49:21 (1781005761) [ 8929.350765] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 8930.821818] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 07:49:24 (1781005764) [ 8934.387241] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 8961.991462] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8962.252699] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8962.288568] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8962.438200] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8966.525769] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 8974.499915] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8974.642687] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8974.672664] Lustre: 291325:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8979.131132] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 9000.302431] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9000.561282] Lustre: Denying initial registration attempt from nid 192.168.203.154@tcp, specified as failover [ 9000.567839] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9000.573493] LustreError: 292749:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -99 [ 9000.582165] LustreError: 292749:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 9000.591071] LustreError: 292749:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9000.719496] Lustre: server umount lustre-OST0000 complete [ 9000.721992] LustreError: 292749:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9010.657509] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9010.676299] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9010.694585] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9015.347211] Lustre: server umount lustre-MDT0000 complete [ 9018.339588] LustreError: 291335:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9018.357810] LustreError: 291335:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9019.479403] LustreError: 291318:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781005854 with bad export cookie 14863485809946662876 [ 9019.483337] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9019.491645] LustreError: 291318:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9019.803183] Lustre: server umount lustre-MDT0001 complete [ 9030.177193] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 9032.362871] Key type lgssc unregistered [ 9032.716620] LNet: 293852:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9032.727778] LNetError: 293852:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9032.753216] LNet: Removed LNI 192.168.203.154@tcp [ 9033.551566] Key type .llcrypt unregistered [ 9033.553490] Key type ._llcrypt unregistered [ 9043.521115] Key type ._llcrypt registered [ 9043.524139] Key type .llcrypt registered [ 9043.626237] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 9044.386575] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9044.482649] alg: No test for adler32 (adler32-zlib) [ 9045.526760] Lustre: Lustre: Build Version: 2.17.54_1_g695b805 [ 9045.806583] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 9047.530469] Key type lgssc registered [ 9048.370473] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9089.616345] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 9096.614380] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9096.627177] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9097.802156] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9097.818953] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9097.922552] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9101.453876] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 9108.740181] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9108.809978] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9108.844238] Lustre: 296561:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9109.080230] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9112.756273] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 9117.831886] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9121.143513] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9139.947717] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9140.167826] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9140.389304] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9145.800194] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 9150.977362] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9154.019270] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9154.027750] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9154.044099] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9154.534703] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9154.536747] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9154.548351] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9159.062969] Lustre: server umount lustre-OST0000 complete [ 9164.770226] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9164.775483] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9164.780327] Lustre: Skipped 1 previous similar message [ 9168.910858] Lustre: server umount lustre-MDT0000 complete [ 9172.461506] LustreError: 296600:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781006007 with bad export cookie 16958109504365917868 [ 9172.471638] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9172.478719] LustreError: 296600:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9172.984468] Lustre: server umount lustre-MDT0001 complete [ 9183.267574] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 9186.338519] Key type lgssc unregistered [ 9186.615228] LNet: 299777:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9186.626063] LNetError: 299777:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9186.651260] LNet: Removed LNI 192.168.203.154@tcp [ 9187.657308] Key type .llcrypt unregistered [ 9187.663459] Key type ._llcrypt unregistered [ 9217.965988] Key type ._llcrypt registered [ 9217.968431] Key type .llcrypt registered [ 9218.051731] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 9219.121222] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9219.187651] alg: No test for adler32 (adler32-zlib) [ 9220.195853] Lustre: Lustre: Build Version: 2.17.54_1_g695b805 [ 9220.575891] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 9222.464158] Key type lgssc registered [ 9224.156094] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9232.132235] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9232.146727] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9233.713756] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9238.231172] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 9243.112918] LustreError: 301177:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9243.153172] LustreError: 301177:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9248.057983] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9248.225736] LustreError: 301178:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9248.511372] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9253.050316] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 9259.473041] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9264.021340] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9273.719751] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9274.259342] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9281.272560] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 9287.654097] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9304.036510] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9304.059592] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9304.070191] Lustre: Skipped 1 previous similar message [ 9305.638379] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.54@tcp (stopping) [ 9305.652548] Lustre: Skipped 2 previous similar messages [ 9308.419161] Lustre: server umount lustre-MDT0000 complete [ 9309.158316] LustreError: 302159:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9309.178363] LustreError: 302159:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9312.089731] LustreError: 301160:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781006146 with bad export cookie 14228499381311100941 [ 9312.090866] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9312.097676] LustreError: 301160:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9314.273266] LustreError: 301172:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9314.277520] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9314.295402] LustreError: 301172:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9314.322616] Lustre: Skipped 1 previous similar message [ 9314.328212] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9317.288608] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9317.294181] Lustre: Skipped 1 previous similar message [ 9318.497658] Lustre: server umount lustre-MDT0001 complete [ 9324.840131] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9333.336926] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9337.845404] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xc575c8cd2de9f579 [ 9337.856996] Lustre: MGC192.168.203.154@tcp: Connection restored to 0@lo (at 0@lo) [ 9338.127299] LustreError: 302785:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9338.144632] LustreError: 302785:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9338.208036] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9338.268328] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 9342.583851] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 9347.434206] LustreError: 302786:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9347.438168] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9347.449381] LustreError: 302786:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9351.495116] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9352.034889] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9356.521524] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 9357.301590] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9362.326644] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9364.067851] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9367.867207] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9369.507933] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9377.761837] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9377.775810] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9377.784673] Lustre: Skipped 1 previous similar message [ 9380.712641] Lustre: server umount lustre-OST0000 complete [ 9388.001735] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9388.006568] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9388.015221] Lustre: Skipped 1 previous similar message [ 9388.023768] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9388.032481] Lustre: Skipped 1 previous similar message [ 9390.410543] Lustre: server umount lustre-MDT0000 complete [ 9393.132168] LustreError: 304461:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9393.152296] LustreError: 304461:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9393.522132] LustreError: 301160:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781006228 with bad export cookie 14228499381311108473 [ 9393.528643] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9393.536858] LustreError: 301160:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9393.816600] Lustre: server umount lustre-MDT0001 complete [ 9401.804610] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 9403.764867] Key type lgssc unregistered [ 9403.988427] LNet: 306886:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9404.000745] LNetError: 306886:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9404.024047] LNet: Removed LNI 192.168.203.154@tcp [ 9404.619541] Key type .llcrypt unregistered [ 9404.622058] Key type ._llcrypt unregistered [ 9422.651947] Key type ._llcrypt registered [ 9422.654353] Key type .llcrypt registered [ 9422.738857] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 9423.561316] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9423.602047] alg: No test for adler32 (adler32-zlib) [ 9424.701717] Lustre: Lustre: Build Version: 2.17.54_1_g695b805 [ 9424.960330] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 9426.592734] Key type lgssc registered [ 9427.779420] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9434.721724] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9434.753314] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9436.217269] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9441.590980] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 9446.378961] LustreError: 308265:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9446.411359] LustreError: 308265:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9451.492421] LustreError: 308264:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9451.561524] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9452.197153] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9456.664248] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 9463.045400] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9467.564844] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9470.809988] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9470.831235] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9476.066990] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9476.074526] Lustre: Skipped 2 previous similar messages [ 9477.091256] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9477.095929] Lustre: Skipped 1 previous similar message [ 9482.215596] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9482.229800] Lustre: Skipped 1 previous similar message [ 9485.280139] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9485.558768] Lustre: server umount lustre-MDT0000 complete [ 9487.330344] LustreError: 308260:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9487.361464] LustreError: 308260:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9489.798958] LustreError: 308244:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781006324 with bad export cookie 6912403963242886232 [ 9489.803174] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9489.806038] LustreError: 308244:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9490.111134] Lustre: server umount lustre-MDT0001 complete [ 9502.836282] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9503.071574] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9503.097517] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9503.278200] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9507.224946] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 9516.149207] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9516.334056] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9516.383962] Lustre: 310628:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9516.653287] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9520.922348] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 9526.265694] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9530.121828] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9539.305351] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9539.492689] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 9539.501694] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 9539.507412] LustreError: 312284:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -2 [ 9539.512496] LustreError: 312284:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 9539.517755] LustreError: 312284:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9539.579926] Lustre: server umount lustre-OST0000 complete [ 9539.588258] LustreError: 312284:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 9551.145869] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9551.306697] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9551.448575] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9558.201451] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 9558.819146] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 9564.876584] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9573.774620] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9573.853464] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 9573.919266] Lustre: lustre-OST0001: new disk, initializing [ 9573.921493] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9573.926166] Lustre: 313717:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9574.049299] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9579.372347] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 9580.681193] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9580.697606] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9580.808461] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9585.039514] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 9591.277220] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9591.279056] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9591.280045] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9591.292257] Lustre: Skipped 2 previous similar messages [ 9591.305942] LustreError: Skipped 1 previous similar message [ 9593.934324] Lustre: server umount lustre-OST0001 complete [ 9596.405047] LustreError: 312753:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9596.426995] LustreError: 312753:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9599.457402] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9599.462679] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9599.472083] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9599.473964] Lustre: Skipped 1 previous similar message [ 9601.510131] LustreError: 312751:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9601.525013] LustreError: 312751:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9611.232332] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9611.437217] Lustre: server umount lustre-OST0000 complete [ 9616.865755] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9616.890227] Lustre: Skipped 1 previous similar message [ 9616.901205] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9616.904243] Lustre: Skipped 5 previous similar messages [ 9621.434190] Lustre: server umount lustre-MDT0000 complete [ 9621.986600] LustreError: 311555:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9622.011417] LustreError: 311555:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9625.528127] LustreError: 313716:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781006460 with bad export cookie 6912403963242886778 [ 9625.532996] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9625.533532] LustreError: 313716:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9625.734638] Lustre: server umount lustre-MDT0001 complete [ 9639.199157] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 9643.272768] Key type lgssc unregistered [ 9643.706689] LNet: 315562:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9643.720930] LNetError: 315562:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9644.777795] LNet: Removed LNI 192.168.203.154@tcp [ 9646.113981] Key type .llcrypt unregistered [ 9646.118325] Key type ._llcrypt unregistered [ 9717.276096] Key type ._llcrypt registered [ 9717.277763] Key type .llcrypt registered [ 9717.372717] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_hostid [ 9731.255632] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 9732.281641] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9732.298178] alg: No test for adler32 (adler32-zlib) [ 9733.258348] Lustre: Lustre: Build Version: 2.17.54_1_g695b805 [ 9733.418719] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 9735.088493] Key type lgssc registered [ 9736.107677] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9777.544537] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9777.572664] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9778.832152] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9778.881773] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9778.959333] Lustre: lustre-MDT0000: new disk, initializing [ 9779.096400] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9779.128121] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9783.474674] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 9794.685513] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9794.871440] Lustre: 320463:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9795.023340] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9795.044747] Lustre: Skipped 1 previous similar message [ 9795.379153] Lustre: lustre-MDT0001: new disk, initializing [ 9795.528241] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9795.566102] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9795.582170] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9801.561638] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 9810.314438] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9814.858913] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9821.132560] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9821.414071] Lustre: lustre-OST0000: new disk, initializing [ 9821.418419] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9821.423749] Lustre: 322207:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9821.499391] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9828.221841] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 9829.460460] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9829.465057] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9829.589271] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9835.045755] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9839.482930] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9839.658833] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9843.827177] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9844.048599] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9850.341589] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9850.344493] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9850.346561] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9850.355694] LustreError: Skipped 1 previous similar message [ 9850.366779] Lustre: Skipped 1 previous similar message [ 9852.272481] Lustre: server umount lustre-OST0000 complete [ 9855.461479] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9855.469407] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9855.471442] Lustre: Skipped 1 previous similar message [ 9855.483890] Lustre: Skipped 1 previous similar message [ 9860.583109] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9860.592465] Lustre: Skipped 2 previous similar messages [ 9861.562096] Lustre: server umount lustre-MDT0000 complete [ 9865.651299] LustreError: 320458:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781006700 with bad export cookie 13454310832632627284 [ 9865.661925] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9865.662835] LustreError: 320458:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9866.186787] Lustre: server umount lustre-MDT0001 complete [ 9875.058130] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 08:05:08 (1781006708) [ 9881.122583] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 08:05:14 (1781006714) [ 9887.020084] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9887.362950] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9890.145512] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 9892.836646] LustreError: 324815:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9892.849866] LustreError: 324815:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9896.971894] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9897.329127] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9900.862446] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 9905.842521] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9910.422187] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9916.165362] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9916.437024] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9920.749138] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 9926.055511] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9943.236656] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 9944.939074] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 9946.387337] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 9947.994701] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 9949.614478] Lustre: Failing over lustre-MDT0000 [ 9949.674122] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9949.679757] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9949.687394] Lustre: Skipped 2 previous similar messages [ 9949.696070] Lustre: Skipped 2 previous similar messages [ 9949.869710] Lustre: server umount lustre-MDT0000 complete [ 9953.302570] LustreError: 324810:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9954.789611] LustreError: 324809:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9954.804252] LustreError: 324809:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9958.413412] LustreError: 325591:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9963.530357] LustreError: 324810:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9963.544900] LustreError: 324810:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9966.675565] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9966.760179] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9966.970465] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9967.018489] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9968.646894] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9970.003261] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 9972.206724] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9972.228772] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 9972.257150] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 9976.026767] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9977.260973] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9980.034983] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 9981.339795] Lustre: Failing over lustre-MDT0000 [ 9981.539944] Lustre: server umount lustre-MDT0000 complete [ 9982.435123] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9982.435920] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9982.440305] Lustre: Skipped 1 previous similar message [ 9982.441227] LustreError: 324810:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9982.462031] LustreError: 324810:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 9998.819681] Lustre: 318689:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781006817/real 1781006817] req@0000000047821f3f x1867520743132416/t0(0) o400->MGC192.168.203.154@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1781006833 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9998.871654] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9998.886769] LustreError: 325525:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9998.906987] LustreError: 325525:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 14 previous similar messages [ 9999.227732] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10009.062853] LustreError: 318687:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000062b29822 x1867520743138432/t0(0) o250->MGC192.168.203.154@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10009.367246] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10009.425284] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10009.683055] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10013.542993] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [10014.727258] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10014.742293] Lustre: Skipped 2 previous similar messages [10014.770961] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [10014.846610] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [10020.933609] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10022.566790] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10030.049352] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10030.051737] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10030.077027] Lustre: Skipped 2 previous similar messages [10035.482492] Lustre: server umount lustre-MDT0000 complete [10039.989113] LustreError: 324795:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781006874 with bad export cookie 13454310832632630000 [10039.997297] LustreError: 324795:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10040.010630] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10040.393917] Lustre: server umount lustre-MDT0001 complete [10055.539681] Lustre: server umount lustre-OST0000 complete [10070.293488] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10070.854161] LustreError: 330617:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10070.869764] LustreError: 330617:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [10074.949734] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [10084.201714] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10084.516776] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10084.524299] Lustre: Skipped 1 previous similar message [10090.233190] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [10097.502356] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10102.999191] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10110.448489] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10115.531688] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [10116.924524] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [10121.171436] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10137.569611] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10137.574955] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10137.585820] Lustre: Skipped 1 previous similar message [10137.597891] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10137.600751] Lustre: Skipped 5 previous similar messages [10139.624730] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10142.868615] Lustre: server umount lustre-OST0000 complete [10154.530294] Lustre: server umount lustre-MDT0000 complete [10154.986039] LustreError: 330617:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10155.022822] LustreError: 330617:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10159.451274] LustreError: 330598:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781006994 with bad export cookie 13454310832632630742 [10159.465846] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10159.466907] LustreError: 330598:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [10159.768320] Lustre: server umount lustre-MDT0001 complete [10174.958800] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [10180.567913] Key type lgssc unregistered [10181.156777] LNet: 333816:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10181.180651] LNetError: 333816:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10181.214542] LNet: Removed LNI 192.168.203.154@tcp [10182.950472] Key type .llcrypt unregistered [10182.954653] Key type ._llcrypt unregistered [10211.301363] Key type ._llcrypt registered [10211.303327] Key type .llcrypt registered [10211.405922] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [10212.268524] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10212.295112] alg: No test for adler32 (adler32-zlib) [10213.292661] Lustre: Lustre: Build Version: 2.17.54_1_g695b805 [10213.466423] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [10215.120549] Key type lgssc registered [10216.380750] Lustre: Echo OBD driver; http://www.lustre.org/ [10225.053979] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10225.077190] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10226.680719] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10231.102309] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [10236.901970] LustreError: 335316:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10236.930475] LustreError: 335316:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10240.097677] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10240.441633] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10244.146231] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [10249.569798] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10252.954385] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10259.847696] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10260.208033] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10266.095526] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [10271.476328] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [10272.112693] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10281.442818] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10281.452251] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10281.462590] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10281.958576] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10281.976255] Lustre: Skipped 1 previous similar message [10281.985207] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10282.000758] Lustre: Skipped 1 previous similar message [10287.006267] Lustre: server umount lustre-MDT0000 complete [10287.077885] LustreError: 335317:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10287.120260] LustreError: 335317:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10290.565662] LustreError: 335297:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781007125 with bad export cookie 4680784175297914788 [10290.575181] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10290.579318] LustreError: 335297:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10290.831978] Lustre: server umount lustre-MDT0001 complete [10305.528910] Lustre: server umount lustre-OST0000 complete [10322.089639] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10322.584963] LustreError: 338688:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10322.695212] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10327.519590] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [10328.037088] LustreError: 338689:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10332.130777] LustreError: 338688:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10336.737690] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10337.142756] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10341.582433] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [10347.335556] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10350.818459] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10358.460030] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10358.950930] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10364.593457] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [10367.176328] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [10372.259320] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10383.331163] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10383.339059] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10383.351358] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10387.939179] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10387.946553] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10387.969456] Lustre: Skipped 1 previous similar message [10387.988115] Lustre: Skipped 2 previous similar messages [10389.565716] Lustre: server umount lustre-MDT0000 complete [10393.062530] LustreError: 340304:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10393.090430] LustreError: 340304:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10393.532555] LustreError: 338668:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781007228 with bad export cookie 4680784175297915908 [10393.556842] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10393.558283] LustreError: 338668:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10393.891442] Lustre: server umount lustre-MDT0001 complete [10408.573240] Lustre: server umount lustre-OST0000 complete [10422.927955] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10423.402464] LustreError: 342263:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10423.477991] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10427.509478] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [10437.443779] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10442.370854] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [10448.819611] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10453.779070] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10464.213259] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10464.663727] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10464.671211] Lustre: Skipped 1 previous similar message [10468.799639] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [10472.479338] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [10479.559765] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10488.295893] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10488.299718] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10488.302785] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10488.334980] Lustre: Skipped 1 previous similar message [10490.720520] Lustre: server umount lustre-OST0000 complete [10498.530084] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10498.543609] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10498.558044] Lustre: Skipped 1 previous similar message [10498.582141] Lustre: Skipped 2 previous similar messages [10500.907312] Lustre: server umount lustre-MDT0000 complete [10503.650639] LustreError: 342263:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10503.687281] LustreError: 342263:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10505.188583] LustreError: 342243:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781007340 with bad export cookie 4680784175297916930 [10505.202523] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10505.551679] Lustre: server umount lustre-MDT0001 complete [10516.484510] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [10519.048074] Key type lgssc unregistered [10519.331389] LNet: 345439:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10519.338949] LNetError: 345439:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10519.353342] LNet: Removed LNI 192.168.203.154@tcp [10520.410519] Key type .llcrypt unregistered [10520.414265] Key type ._llcrypt unregistered [10548.194356] Key type ._llcrypt registered [10548.196886] Key type .llcrypt registered [10548.324798] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [10549.645428] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10549.781438] alg: No test for adler32 (adler32-zlib) [10550.899099] Lustre: Lustre: Build Version: 2.17.54_1_g695b805 [10551.177454] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [10552.872842] Key type lgssc registered [10554.004348] Lustre: Echo OBD driver; http://www.lustre.org/ [10560.983360] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10561.006819] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10561.018288] LustreError: 346864:0:(osd_handler.c:8551:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [10561.047884] LustreError: 346864:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [10561.055926] LustreError: 346864:0:(obd_mount.c:250:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [10561.066092] LustreError: 346864:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [10561.077431] LustreError: 346864:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [10567.712570] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [10567.721575] LustreError: 347261:0:(osd_handler.c:8551:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [10567.738497] LustreError: 347261:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [10567.742623] LustreError: 347261:0:(obd_mount.c:250:lustre_start_simple()) lustre-OST0000-osd setup error -22 [10567.746397] LustreError: 347261:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [10567.751047] LustreError: 347261:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [10588.195424] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [10592.401382] Key type lgssc unregistered [10592.900052] LNet: 348248:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10592.914450] LNetError: 348248:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10592.940320] LNet: Removed LNI 192.168.203.154@tcp [10594.504701] Key type .llcrypt unregistered [10594.506928] Key type ._llcrypt unregistered [10616.758198] Key type ._llcrypt registered [10616.761731] Key type .llcrypt registered [10616.892744] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_hostid [10632.813333] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [10633.951595] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10634.024066] alg: No test for adler32 (adler32-zlib) [10635.100630] Lustre: Lustre: Build Version: 2.17.54_1_g695b805 [10635.424704] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [10637.072258] Key type lgssc registered [10638.038761] Lustre: Echo OBD driver; http://www.lustre.org/ [10676.959981] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10676.988861] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10678.239410] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10678.271940] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10678.318732] Lustre: lustre-MDT0000: new disk, initializing [10678.460682] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10678.489527] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10683.000891] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [10695.371423] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10695.486909] Lustre: 351370:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10695.575163] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10695.581696] Lustre: Skipped 1 previous similar message [10695.689398] Lustre: lustre-MDT0001: new disk, initializing [10695.857123] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10695.911876] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10695.931839] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10701.023730] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [10711.136965] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10714.642723] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10721.893873] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10722.249403] Lustre: lustre-OST0000: new disk, initializing [10722.255273] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10722.265268] Lustre: 353114:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10722.458155] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10728.018105] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10728.042995] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10728.162852] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10728.742843] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [10737.105111] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10742.052614] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10742.312208] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10747.493467] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10747.764579] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10754.023809] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10754.040984] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10754.059365] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10756.404710] Lustre: server umount lustre-OST0000 complete [10762.724734] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10762.746394] LustreError: Skipped 1 previous similar message [10762.751415] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10762.775366] Lustre: Skipped 1 previous similar message [10762.784956] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10762.795439] Lustre: Skipped 1 previous similar message [10764.261392] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10764.264095] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10766.997204] Lustre: server umount lustre-MDT0000 complete [10770.862842] LustreError: 351364:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781007605 with bad export cookie 6857975592350514568 [10770.866582] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10770.872374] LustreError: 351364:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10771.383481] Lustre: server umount lustre-MDT0001 complete [10783.125100] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [10785.643464] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 08:20:18 (1781007618) [10792.587270] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10792.905731] LustreError: 355425:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10792.926340] LustreError: 355425:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10792.994990] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10797.329483] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [10798.051469] LustreError: 355426:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10802.149863] LustreError: 355425:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10805.983672] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10806.368383] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10810.884476] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [10815.929971] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10820.095409] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10826.805120] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10827.134383] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10832.753941] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [10838.347752] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10844.769373] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10844.920475] Lustre: lustre-OST0001: new disk, initializing [10844.927449] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10844.939889] Lustre: 357924:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10845.062878] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10850.419193] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10850.428295] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10850.523873] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10851.125312] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [10858.479076] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [10865.229247] Lustre: lustre-OST0001: Not available for connect from 192.168.203.54@tcp (stopping) [10865.237672] Lustre: Skipped 1 previous similar message [10866.146370] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10866.149540] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10866.157020] LustreError: Skipped 1 previous similar message [10866.202489] Lustre: Skipped 1 previous similar message [10868.140995] Lustre: server umount lustre-OST0001 complete [10870.281708] LustreError: 357021:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10876.417201] LustreError: 357021:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10876.436706] LustreError: 357021:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10886.630263] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10886.635989] LustreError: 357021:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10886.641836] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10886.641847] Lustre: Skipped 2 previous similar messages [10886.672649] Lustre: Skipped 1 previous similar message [10886.711912] LustreError: 357021:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10892.112484] Lustre: server umount lustre-OST0000 complete [10898.405207] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10898.428925] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10898.457608] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10898.465403] Lustre: Skipped 1 previous similar message [10904.727467] Lustre: server umount lustre-MDT0000 complete [10907.128559] LustreError: 356137:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10907.201206] LustreError: 356137:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [10909.667240] LustreError: 355407:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781007744 with bad export cookie 6857975592350515450 [10909.677863] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10909.697051] LustreError: 355407:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10916.647516] Lustre: server umount lustre-MDT0001 complete [10928.393934] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [10931.365150] Key type lgssc unregistered [10931.748977] LNet: 359781:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10931.754840] LNetError: 359781:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10931.774876] LNet: Removed LNI 192.168.203.154@tcp [10932.857578] Key type .llcrypt unregistered [10932.864527] Key type ._llcrypt unregistered [10988.438675] Key type ._llcrypt registered [10988.442403] Key type .llcrypt registered [10988.560824] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [10989.460802] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10989.585132] alg: No test for adler32 (adler32-zlib) [10990.649921] Lustre: Lustre: Build Version: 2.17.54_1_g695b805 [10990.834102] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [10992.491460] Key type lgssc registered [10993.573326] Lustre: Echo OBD driver; http://www.lustre.org/ [11001.149066] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11001.177185] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11002.425048] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [11002.447773] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11002.581965] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11007.806577] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [11015.971895] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11016.091030] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11016.141472] Lustre: 362568:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11016.432922] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11020.828481] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [11026.686460] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11031.431841] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11037.154502] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11037.160449] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11037.172313] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11038.179650] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11038.181336] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11040.236962] Lustre: server umount lustre-MDT0000 complete [11044.252293] LustreError: 362560:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781007879 with bad export cookie 17683669095308405552 [11044.262364] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11044.269941] LustreError: 362560:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11044.721257] Lustre: server umount lustre-MDT0001 complete [11056.436433] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 08:24:48 (1781007888) [11063.756801] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11064.228722] LustreError: 364854:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11064.252545] LustreError: 364854:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [11064.269298] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11068.071208] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [11069.409464] LustreError: 364855:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11074.529079] LustreError: 364854:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11078.104612] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11078.698084] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11084.296380] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [11091.024821] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11095.297237] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11103.100849] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11103.303942] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [11103.550289] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11110.124946] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [11116.562396] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11126.145143] Lustre: Permanently deactivating lustre-OST0000 [11126.181539] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [11136.030627] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11139.135786] LustreError: 367548:0:(mgs_llog.c:2041:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [11139.167974] LustreError: 367548:0:(mgs_llog.c:2096:mgs_replace_nids()) Only MGS is allowed to be started [11139.183230] LustreError: 367548:0:(mgs_handler.c:1244:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11143.691051] LustreError: 367597:0:(mgs_llog.c:2096:mgs_replace_nids()) Only MGS is allowed to be started [11143.695764] LustreError: 367597:0:(mgs_handler.c:1244:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11151.912915] Lustre: server umount lustre-OST0000 complete [11155.444570] LustreError: 367847:0:(mgs_llog.c:2096:mgs_replace_nids()) Only MGS is allowed to be started [11155.449939] LustreError: 367847:0:(mgs_handler.c:1244:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11160.548263] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11160.555269] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11160.563417] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11160.566159] Lustre: Skipped 1 previous similar message [11167.201630] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11167.218366] Lustre: Skipped 3 previous similar messages [11167.301617] Lustre: server umount lustre-MDT0000 complete [11171.463929] LustreError: 364835:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781008006 with bad export cookie 17683669095308406133 [11171.466969] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11171.472664] LustreError: 364835:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11182.272512] Lustre: server umount lustre-MDT0001 complete [11191.640471] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11196.754721] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [11199.914597] LustreError: 368991:0:(mgs_handler.c:1244:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [11207.458722] Lustre: 369330:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.154@tcp [11209.268098] Lustre: server umount MGS complete [11221.165771] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11221.487279] LustreError: 370053:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11221.497173] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11221.510382] Lustre: Skipped 1 previous similar message [11221.551880] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11226.103651] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [11226.597450] LustreError: 370054:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11234.883184] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11235.160064] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11239.626569] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [11245.338748] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11249.241356] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11251.215540] Lustre: Permanently reactivating lustre-OST0000 [11251.246447] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [11251.250205] Lustre: Skipped 2 previous similar messages [11280.864162] LustreError: 371655:0:(osp_dev.c:693:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [11288.025235] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11288.399309] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11288.416825] Lustre: Skipped 1 previous similar message [11294.466501] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [11300.200702] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11305.955882] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11305.960907] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11305.970502] LustreError: Skipped 1 previous similar message [11305.971676] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11306.003734] Lustre: Skipped 2 previous similar messages [11307.825823] Lustre: server umount lustre-OST0000 complete [11312.098063] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11312.103670] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11314.157325] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11314.169188] Lustre: Skipped 3 previous similar messages [11317.747706] Lustre: server umount lustre-MDT0000 complete [11319.272905] LustreError: 370765:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11319.302258] LustreError: 370765:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11321.359872] LustreError: 370034:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781008156 with bad export cookie 17683669095308407183 [11321.361286] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11321.375140] LustreError: 370034:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11321.940422] Lustre: server umount lustre-MDT0001 complete [11331.036382] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [11333.246554] Key type lgssc unregistered [11333.552863] LNet: 374157:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11333.562527] LNetError: 374157:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11333.601383] LNet: Removed LNI 192.168.203.154@tcp [11334.250985] Key type .llcrypt unregistered [11334.252714] Key type ._llcrypt unregistered [11351.770724] Key type ._llcrypt registered [11351.775949] Key type .llcrypt registered [11351.891062] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_hostid [11366.073284] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [11366.864586] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11366.935232] alg: No test for adler32 (adler32-zlib) [11367.968336] Lustre: Lustre: Build Version: 2.17.54_1_g695b805 [11368.201029] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [11369.832180] Key type lgssc registered [11370.764791] Lustre: Echo OBD driver; http://www.lustre.org/ [11400.412478] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11400.434648] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11401.731344] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11401.773852] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11401.898706] Lustre: lustre-MDT0000: new disk, initializing [11401.976674] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11402.021162] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11405.778913] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [11415.412538] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11415.515104] Lustre: 377281:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11415.563859] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11415.569079] Lustre: Skipped 1 previous similar message [11415.629198] Lustre: lustre-MDT0001: new disk, initializing [11415.719156] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11415.770569] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11415.782544] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11419.077454] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [11425.245343] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11428.839880] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11435.314983] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11435.580495] Lustre: lustre-OST0000: new disk, initializing [11435.583622] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11435.589901] Lustre: 379022:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11435.676225] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11440.776865] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [11441.207640] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11441.216733] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11441.314958] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11447.873917] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11451.880749] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11452.183748] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11456.823565] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11457.108880] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11461.615776] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11461.617729] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11461.627360] Lustre: Skipped 1 previous similar message [11466.734588] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11466.739856] Lustre: Skipped 2 previous similar messages [11471.848814] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11471.853377] Lustre: Skipped 1 previous similar message [11473.376187] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11473.476477] Lustre: server umount lustre-OST0000 complete [11476.961965] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11476.964184] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11476.969066] Lustre: Skipped 1 previous similar message [11476.978709] Lustre: Skipped 1 previous similar message [11481.903345] Lustre: server umount lustre-MDT0000 complete [11484.303941] LustreError: 378068:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781008319 with bad export cookie 9125808364051332166 [11484.309273] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11484.314357] LustreError: 378068:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11484.509261] Lustre: server umount lustre-MDT0001 complete [11490.249288] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [11491.394959] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 08:32:05 (1781008325) [11495.922037] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11496.229865] LustreError: 381335:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11496.245895] LustreError: 381335:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [11496.285792] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11499.157417] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [11501.539031] LustreError: 381336:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11505.126291] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11505.432892] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11508.267834] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [11512.117077] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11514.909252] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11519.766625] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11519.939383] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11524.166608] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [11528.022568] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11529.234621] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [11531.288543] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [11535.329875] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11535.331024] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11535.338658] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11535.350305] Lustre: Skipped 1 previous similar message [11540.836953] Lustre: server umount lustre-OST0000 complete [11545.577342] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11545.583396] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11545.589931] Lustre: Skipped 1 previous similar message [11545.594608] Lustre: Skipped 3 previous similar messages [11549.574879] Lustre: server umount lustre-MDT0000 complete [11550.689624] LustreError: 381331:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11550.699662] LustreError: 381331:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11551.946424] LustreError: 381316:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781008386 with bad export cookie 9125808364051333048 [11551.948436] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11551.955573] LustreError: 381316:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11552.248765] Lustre: server umount lustre-MDT0001 complete [11560.122399] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [11561.945181] Key type lgssc unregistered [11562.129362] LNet: 384699:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11562.137061] LNetError: 384699:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11562.151178] LNet: Removed LNI 192.168.203.154@tcp [11562.601148] Key type .llcrypt unregistered [11562.603068] Key type ._llcrypt unregistered [11578.431630] Key type ._llcrypt registered [11578.433306] Key type .llcrypt registered [11578.504362] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [11578.710415] Key type .llcrypt unregistered [11578.713865] Key type ._llcrypt unregistered [11592.518416] Key type ._llcrypt registered [11592.521704] Key type .llcrypt registered [11592.603266] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [11593.487345] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11593.507480] alg: No test for adler32 (adler32-zlib) [11594.470365] Lustre: Lustre: Build Version: 2.17.54_1_g695b805 [11594.683026] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [11596.312179] Key type lgssc registered [11597.212780] Lustre: Echo OBD driver; http://www.lustre.org/ [11602.642226] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11602.651741] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11603.962492] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11606.907582] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [11612.502855] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11612.743154] LustreError: 386856:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11612.754592] LustreError: 386856:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [11612.762202] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11613.807060] LustreError: 386850:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11613.816694] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [11613.825356] LustreError: 386850:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11616.596796] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [11619.299257] LustreError: 386855:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11619.316505] LustreError: 386855:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11620.493433] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11624.422187] LustreError: 386856:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11624.442053] LustreError: 386856:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11624.953639] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11625.209092] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11627.503858] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [11639.784694] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11639.797189] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11639.806675] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11640.801878] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11640.805169] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11640.811436] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11643.130534] Lustre: server umount lustre-OST0000 complete [11645.512682] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [11645.518502] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11645.529767] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11645.534210] Lustre: Skipped 1 previous similar message [11651.042287] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11651.047838] Lustre: Skipped 2 previous similar messages [11653.688542] Lustre: server umount lustre-MDT0000 complete [11655.862910] LustreError: 386836:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781008490 with bad export cookie 7769656622401793236 [11655.867108] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11655.871214] LustreError: 386836:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11656.076272] Lustre: server umount lustre-MDT0001 complete [11662.389945] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [11664.123692] Key type lgssc unregistered [11664.300562] LNet: 389837:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11664.304819] LNetError: 389837:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11664.314669] LNet: Removed LNI 192.168.203.154@tcp [11664.929584] Key type .llcrypt unregistered [11664.933780] Key type ._llcrypt unregistered [11680.898063] Key type ._llcrypt registered [11680.901074] Key type .llcrypt registered [11680.980110] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [11681.592741] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11681.661638] alg: No test for adler32 (adler32-zlib) [11682.661993] Lustre: Lustre: Build Version: 2.17.54_1_g695b805 [11682.798066] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [11684.432143] Key type lgssc registered [11685.173733] Lustre: Echo OBD driver; http://www.lustre.org/ [11690.900281] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11690.913886] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11706.402111] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [11711.520198] LustreError: 391205:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.154@tcp: failed processing log, type 4: rc = -110 [11738.144411] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11748.463169] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [11756.047953] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11767.666336] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11768.496680] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11768.596870] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [11775.061359] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [11778.026957] LustreError: 392130:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11778.044677] LustreError: 392130:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [11783.139825] LustreError: 392131:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11783.168118] LustreError: 392131:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11788.264715] LustreError: 392130:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11788.301890] LustreError: 392130:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11789.073618] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11789.842354] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11795.686241] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [11802.740411] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11807.656181] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11819.491090] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11819.502123] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11819.541055] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11820.521513] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11820.523631] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11820.541806] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11823.691713] Lustre: server umount lustre-OST0000 complete [11828.653442] Lustre: server umount lustre-MDT0000 complete [11830.756260] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11830.759080] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11830.765295] LustreError: 392131:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11830.818017] Lustre: Skipped 1 previous similar message [11832.928087] LustreError: 391212:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781008667 with bad export cookie 6740363338164960268 [11832.942455] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11833.399213] Lustre: server umount lustre-MDT0001 complete [11846.007436] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [11848.639635] Key type lgssc unregistered [11848.881298] LNet: 394419:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11848.895376] LNetError: 394419:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11848.911736] LNet: Removed LNI 192.168.203.154@tcp [11849.884340] Key type .llcrypt unregistered [11849.887246] Key type ._llcrypt unregistered [11875.741205] Key type ._llcrypt registered [11875.743966] Key type .llcrypt registered [11875.862775] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [11876.550796] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11876.596938] alg: No test for adler32 (adler32-zlib) [11877.638323] Lustre: Lustre: Build Version: 2.17.54_1_g695b805 [11877.923505] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [11879.627741] Key type lgssc registered [11880.556737] Lustre: Echo OBD driver; http://www.lustre.org/ [11887.828177] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11887.848740] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11889.283231] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11893.232298] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [11899.878031] LustreError: 395819:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11899.905645] LustreError: 395819:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11902.238773] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11902.870545] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11908.638889] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [11915.770657] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11920.351227] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11928.658099] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11929.107973] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11934.262941] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [11935.028287] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [11941.240727] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11948.069237] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11948.079510] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11948.092049] Lustre: Skipped 2 previous similar messages [11948.109798] Lustre: Skipped 3 previous similar messages [11951.333743] Lustre: server umount lustre-MDT0000 complete [11953.129328] LustreError: 395819:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11953.152242] LustreError: 395819:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11953.968680] LustreError: 395801:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.54@tcp arrived at 1781008788 with bad export cookie 5719158508742554786 [11956.058622] LustreError: 396535:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781008790 with bad export cookie 5719158508742553897 [11956.072828] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11956.078823] LustreError: 396535:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11958.242829] LustreError: 395814:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11958.245675] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11958.262022] LustreError: 395814:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11958.295257] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11961.256325] LustreError: 395819:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11961.271527] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11961.279674] LustreError: 395819:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11962.274648] Lustre: server umount lustre-OST0000 complete [11965.346544] LustreError: 395818:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11965.364719] LustreError: 395818:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11969.528381] LustreError: 396530:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000553c47f2 x1867522991226496/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11969.823479] Lustre: server umount lustre-MDT0001 complete [11980.168780] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [11982.974453] Key type lgssc unregistered [11983.281720] LNet: 399147:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11983.287349] LNetError: 399147:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11983.308954] LNet: Removed LNI 192.168.203.154@tcp [11984.230938] Key type .llcrypt unregistered [11984.235348] Key type ._llcrypt unregistered [12009.850596] Key type ._llcrypt registered [12009.851975] Key type .llcrypt registered [12009.945385] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [12011.001732] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12011.044429] alg: No test for adler32 (adler32-zlib) [12012.343882] Lustre: Lustre: Build Version: 2.17.54_1_g695b805 [12012.577277] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [12014.200166] Key type lgssc registered [12016.269581] Lustre: Echo OBD driver; http://www.lustre.org/ [12026.801750] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12026.836987] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12028.519958] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12032.532521] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [12038.128885] LustreError: 400549:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12038.155397] LustreError: 400549:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12041.534557] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12041.994720] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12045.726670] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [12051.545593] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12057.480886] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12065.069959] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12065.506848] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12072.658512] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [12073.731564] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [12080.530213] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12087.430640] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.54@tcp (stopping) [12088.033485] Lustre: server umount lustre-MDT0001 complete [12088.290803] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12088.315772] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12088.398547] LustreError: 400549:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12089.353028] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12089.413066] LustreError: 400544:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12089.460237] LustreError: 400544:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12094.452238] LustreError: 400549:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12094.485363] LustreError: 400549:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12099.556333] LustreError: 400548:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12099.574433] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12099.588483] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12099.669207] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12104.115670] Lustre: server umount lustre-OST0000 complete [12109.796213] LustreError: 400549:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12109.807873] LustreError: 400549:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12111.029637] Lustre: server umount lustre-MDT0000 complete [12135.598827] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [12139.587835] Key type lgssc unregistered [12140.087580] LNet: 403900:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12140.099472] LNetError: 403900:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12140.116775] LNet: Removed LNI 192.168.203.154@tcp [12141.150887] Key type .llcrypt unregistered [12141.156251] Key type ._llcrypt unregistered [12174.072270] Key type ._llcrypt registered [12174.076215] Key type .llcrypt registered [12174.251670] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_hostid [12193.156381] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [12194.795037] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12194.846089] alg: No test for adler32 (adler32-zlib) [12195.996676] Lustre: Lustre: Build Version: 2.17.54_1_g695b805 [12196.182795] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [12197.912220] Key type lgssc registered [12199.293437] Lustre: Echo OBD driver; http://www.lustre.org/ [12244.492602] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [12262.007680] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12262.043290] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12263.502279] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12263.563577] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12263.659542] Lustre: lustre-MDT0000: new disk, initializing [12263.754565] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12263.777204] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12269.650142] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [12287.928771] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12288.119449] Lustre: lustre-OST0000: new disk, initializing [12288.122584] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12288.129739] Lustre: Skipped 1 previous similar message [12288.137311] Lustre: 408882:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12288.208823] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12293.466111] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [12295.238319] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [12295.250176] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [12295.285883] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [12300.914503] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12317.387587] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12317.549383] Lustre: 407882:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12317.614642] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12317.690641] Lustre: lustre-MDT0001: new disk, initializing [12317.790860] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12317.826870] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [12317.839689] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [12322.298839] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [12332.467544] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [12332.768777] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [12337.824384] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [12338.120813] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [12344.810733] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12344.811849] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12344.840634] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12344.849178] Lustre: Skipped 1 previous similar message [12349.507963] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.54@tcp (stopping) [12354.577470] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.54@tcp (stopping) [12354.593156] Lustre: Skipped 2 previous similar messages [12359.136135] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12359.489609] Lustre: server umount lustre-MDT0001 complete [12364.821773] LustreError: 407887:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12364.849549] LustreError: 407887:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [12370.421159] LustreError: 407889:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12370.453145] LustreError: 407889:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [12375.528021] LustreError: 409554:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12375.551917] LustreError: 409554:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12380.645991] LustreError: 407888:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12380.665880] LustreError: 407888:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12382.188435] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12382.207753] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12382.242504] Lustre: Skipped 1 previous similar message [12382.252906] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12382.266785] Lustre: Skipped 2 previous similar messages [12385.768053] LustreError: 407892:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12387.449904] Lustre: server umount lustre-OST0000 complete [12406.240353] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12406.639864] Lustre: server umount lustre-MDT0000 complete [12426.190666] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [12428.549674] Key type lgssc unregistered [12428.875844] LNet: 412475:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12428.884987] LNetError: 412475:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12428.912429] LNet: Removed LNI 192.168.203.154@tcp [12429.803845] Key type .llcrypt unregistered [12429.807125] Key type ._llcrypt unregistered [12478.802673] Key type ._llcrypt registered [12478.804770] Key type .llcrypt registered [12478.905257] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [12479.143573] Key type .llcrypt unregistered [12479.145775] Key type ._llcrypt unregistered [12498.384680] Key type ._llcrypt registered [12498.387694] Key type .llcrypt registered [12498.497496] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [12499.933487] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12500.021971] alg: No test for adler32 (adler32-zlib) [12501.155277] Lustre: Lustre: Build Version: 2.17.54_1_g695b805 [12501.411253] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [12503.136512] Key type lgssc registered [12504.803265] Lustre: Echo OBD driver; http://www.lustre.org/ [12535.313830] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12535.333518] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12536.801692] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12536.843276] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12536.907161] Lustre: lustre-MDT0000: new disk, initializing [12537.030184] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12537.056534] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12541.944928] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [12552.545606] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12552.649833] Lustre: 415822:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12552.694607] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12552.698635] Lustre: Skipped 1 previous similar message [12552.758967] Lustre: lustre-MDT0001: new disk, initializing [12552.860823] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12552.896974] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12552.907915] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12557.164363] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [12564.923383] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12568.737654] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12575.128339] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12575.431665] Lustre: lustre-OST0000: new disk, initializing [12575.442277] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12575.452994] Lustre: 417561:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12575.542703] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12580.706445] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [12581.988807] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12582.004899] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12582.151155] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12587.377715] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12592.612639] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12592.620792] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12592.621449] Lustre: Skipped 1 previous similar message [12592.632464] Lustre: Skipped 2 previous similar messages [12597.734491] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12598.932787] Lustre: server umount lustre-MDT0000 complete [12602.046374] LustreError: 417571:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781009436 with bad export cookie 2548353036731214791 [12602.052837] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12602.063733] LustreError: 417571:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12602.329330] Lustre: server umount lustre-MDT0001 complete [12615.961667] Lustre: server umount lustre-OST0000 complete [12624.133514] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 08:50:57 (1781009457) [12630.171191] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12630.761314] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12634.465988] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [12636.129472] LustreError: 419475:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12636.148443] LustreError: 419475:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12640.230351] LustreError: 419476:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12641.065920] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12641.353904] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12644.366773] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [12648.664270] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12651.886765] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12670.826949] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12671.088381] Lustre: Found index 0 for lustre-OST0000, updating log [12671.268437] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12675.379183] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [12676.007989] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [12680.267756] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12685.797310] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12685.800334] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12685.824976] Lustre: Skipped 1 previous similar message [12685.836991] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12685.842459] Lustre: Skipped 2 previous similar messages [12687.907068] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12687.914415] Lustre: Skipped 1 previous similar message [12689.825504] Lustre: server umount lustre-OST0000 complete [12704.656111] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12704.658438] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12704.666709] Lustre: Skipped 1 previous similar message [12704.678777] Lustre: Skipped 2 previous similar messages [12710.924660] Lustre: server umount lustre-MDT0000 complete [12713.394343] LustreError: 419457:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781009548 with bad export cookie 2548353036731216191 [12713.397422] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12713.402806] LustreError: 419457:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12713.627087] Lustre: server umount lustre-MDT0001 complete [12719.774165] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 08:52:33 (1781009553) [12740.000461] LNet: 1 local NIs in recovery (showing 1): 192.168.203.154@tcp [13310.558105] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13310.751558] LustreError: 425549:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13310.783482] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13312.307818] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [13315.689689] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13315.840620] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13317.367808] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [13319.657982] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13321.100476] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13337.188443] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13337.307241] Lustre: Found index 0 for lustre-OST0000, updating log [13337.378957] LustreError: 427161:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13337.463709] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13339.699837] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [13341.943571] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13342.807390] Lustre: server umount lustre-OST0000 complete [13346.785346] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13346.789477] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13346.797060] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13346.799779] Lustre: Skipped 4 previous similar messages [13348.833663] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13348.834313] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13350.432503] Lustre: server umount lustre-MDT0000 complete [13351.993074] LustreError: 425529:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781010186 with bad export cookie 2548353036731217262 [13351.997702] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13352.002551] LustreError: 425529:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13352.154928] Lustre: server umount lustre-MDT0001 complete [13365.664459] LNet: 1 local NIs in recovery (showing 1): 192.168.203.154@tcp [13370.922914] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_hostid [13374.809441] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [13389.647879] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 09:03:43 (1781010223) [13396.251578] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [13397.444552] Key type lgssc unregistered [13399.649171] LNet: 431762:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13399.652688] LNetError: 431762:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13469.286484] LNet: Removed LNI 192.168.203.154@tcp [13469.628993] Key type .llcrypt unregistered [13469.630859] Key type ._llcrypt unregistered [13476.768341] Key type ._llcrypt registered [13476.769756] Key type .llcrypt registered [13476.814815] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [13477.150351] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13477.205225] alg: No test for adler32 (adler32-zlib) [13478.080546] Lustre: Lustre: Build Version: 2.17.54_1_g695b805 [13478.178962] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [13479.768397] Key type lgssc registered [13480.200667] Lustre: Echo OBD driver; http://www.lustre.org/ [13481.940062] LNet: Added LNI 192.168.203.154@tcp100 [8/256/0/180] [13482.287528] LNet: Added LNI 192.168.203.154@tcp101 [8/256/0/180] [13482.601413] LNet: Added LNI 192.168.203.154@tcp102 [8/256/0/180] [13482.913769] LNet: Added LNI 192.168.203.154@tcp103 [8/256/0/180] [13483.242054] LNet: Added LNI 192.168.203.154@tcp104 [8/256/0/180] [13483.583754] LNet: Added LNI 192.168.203.154@tcp105 [8/256/0/180] [13483.937742] LNet: Added LNI 192.168.203.154@tcp106 [8/256/0/180] [13484.302642] LNet: Added LNI 192.168.203.154@tcp107 [8/256/0/180] [13484.637442] LNet: Added LNI 192.168.203.154@tcp108 [8/256/0/180] [13484.993538] LNet: Added LNI 192.168.203.154@tcp109 [8/256/0/180] [13485.327061] LNet: Added LNI 192.168.203.154@tcp110 [8/256/0/180] [13485.684868] LNet: Added LNI 192.168.203.154@tcp111 [8/256/0/180] [13486.027881] LNet: Added LNI 192.168.203.154@tcp112 [8/256/0/180] [13486.400126] LNet: Added LNI 192.168.203.154@tcp113 [8/256/0/180] [13486.773636] LNet: Added LNI 192.168.203.154@tcp114 [8/256/0/180] [13487.133527] LNet: Added LNI 192.168.203.154@tcp115 [8/256/0/180] [13487.463581] LNet: Added LNI 192.168.203.154@tcp116 [8/256/0/180] [13487.808096] LNet: Added LNI 192.168.203.154@tcp117 [8/256/0/180] [13488.130712] LNet: Added LNI 192.168.203.154@tcp118 [8/256/0/180] [13488.434663] LNet: Added LNI 192.168.203.154@tcp119 [8/256/0/180] [13488.753484] LNet: Added LNI 192.168.203.154@tcp120 [8/256/0/180] [13489.070972] LNet: Added LNI 192.168.203.154@tcp121 [8/256/0/180] [13489.384433] LNet: Added LNI 192.168.203.154@tcp122 [8/256/0/180] [13489.699659] LNet: Added LNI 192.168.203.154@tcp123 [8/256/0/180] [13490.024229] LNet: Added LNI 192.168.203.154@tcp124 [8/256/0/180] [13490.327287] LNet: Added LNI 192.168.203.154@tcp125 [8/256/0/180] [13490.632341] LNet: Added LNI 192.168.203.154@tcp126 [8/256/0/180] [13490.944726] LNet: Added LNI 192.168.203.154@tcp127 [8/256/0/180] [13491.256217] LNet: Added LNI 192.168.203.154@tcp128 [8/256/0/180] [13491.575263] LNet: Added LNI 192.168.203.154@tcp129 [8/256/0/180] [13491.885543] LNet: Added LNI 192.168.203.154@tcp130 [8/256/0/180] [13492.209756] LNet: Added LNI 192.168.203.154@tcp131 [8/256/0/180] [13492.538624] LNet: Added LNI 192.168.203.154@tcp132 [8/256/0/180] [13492.850711] LNet: Added LNI 192.168.203.154@tcp133 [8/256/0/180] [13493.179198] LNet: Added LNI 192.168.203.154@tcp134 [8/256/0/180] [13493.505185] LNet: Added LNI 192.168.203.154@tcp135 [8/256/0/180] [13496.207343] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13496.218270] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13497.268959] Lustre: 434744:0:(lustre_peer.c:105:class_add_uuid()) 192.168.203.154@tcp: can't add NID 192.168.203.154@tcp130: rc = -75 [13497.301093] LNetError: 434764:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.154 [13497.306045] LNetError: Protocol error connecting to 192.168.203.154@tcp120 at host 192.168.203.154:988: is it running a compatible version of Lustre? [13497.311133] LNetError: Skipped 3 previous similar messages [13497.365924] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13497.383419] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13497.424705] Lustre: lustre-MDT0000: new disk, initializing [13497.465111] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13497.475336] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13497.632085] LNet: 6 local NIs in recovery (showing 6): 192.168.203.154@tcp118, 192.168.203.154@tcp125, 192.168.203.154@tcp120, 192.168.203.154@tcp127, 192.168.203.154@tcp128, 192.168.203.154@tcp129 [13498.338371] LNetError: 434768:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.154 [13498.343828] LNetError: 434768:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 10 previous similar messages [13498.348445] LNetError: Protocol error connecting to 192.168.203.154@tcp112 at host 192.168.203.154:988: is it running a compatible version of Lustre? [13498.353637] LNetError: Skipped 7 previous similar messages [13499.019785] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [13500.708788] LNetError: 434772:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.154 [13500.715113] LNetError: 434772:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 6 previous similar messages [13500.718307] LNetError: Protocol error connecting to 192.168.203.154@tcp125 at host 192.168.203.154:988: is it running a compatible version of Lustre? [13500.722831] LNetError: Skipped 7 previous similar messages [13503.440822] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13503.486717] Lustre: 434761:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13503.513523] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13503.516128] Lustre: Skipped 1 previous similar message [13503.552349] Lustre: lustre-MDT0001: new disk, initializing [13503.564342] Lustre: 435546:0:(lustre_peer.c:105:class_add_uuid()) 192.168.203.154@tcp: can't add NID 192.168.203.154@tcp130: rc = -75 [13503.568042] Lustre: 435546:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [13503.606833] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13503.619355] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13503.623200] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13504.801447] LNetError: 432205:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.154 [13504.805205] LNetError: 432205:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 1 previous similar message [13504.807755] LNetError: Protocol error connecting to 192.168.203.154@tcp125 at host 192.168.203.154:988: is it running a compatible version of Lustre? [13504.948626] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [13508.044919] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13509.182441] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13511.392963] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13511.505324] Lustre: lustre-OST0000: new disk, initializing [13511.507346] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13511.511730] Lustre: 436505:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13511.556599] Lustre: 436510:0:(lustre_peer.c:105:class_add_uuid()) 192.168.203.154@tcp: can't add NID 192.168.203.154@tcp130: rc = -75 [13511.560452] Lustre: 436510:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [13511.581781] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13512.515235] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13512.519300] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13512.590544] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13512.993341] LNetError: 432208:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.154 [13512.997601] LNetError: Protocol error connecting to 192.168.203.154@tcp125 at host 192.168.203.154:988: is it running a compatible version of Lustre? [13513.584990] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [13516.979943] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13522.917233] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13522.919329] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13522.922848] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13522.925279] LustreError: Skipped 1 previous similar message [13522.930188] Lustre: Skipped 1 previous similar message [13527.329422] LNetError: 434764:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.154 [13527.334441] LNetError: Protocol error connecting to 192.168.203.154@tcp123 at host 192.168.203.154:988: is it running a compatible version of Lustre? [13528.037356] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13528.040377] Lustre: Skipped 1 previous similar message [13533.155885] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13533.159042] Lustre: Skipped 1 previous similar message [13534.176203] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13534.250080] Lustre: server umount lustre-OST0000 complete [13538.273728] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13538.274336] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13538.278431] Lustre: Skipped 2 previous similar messages [13543.393954] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13543.397304] Lustre: Skipped 2 previous similar messages [13545.761472] LNetError: 432207:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.154 [13545.771512] LNetError: 432207:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 10 previous similar messages [13545.775304] LNetError: Protocol error connecting to 192.168.203.154@tcp124 at host 192.168.203.154:988: is it running a compatible version of Lustre? [13545.781980] LNetError: Skipped 10 previous similar messages [13550.048147] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13550.134370] Lustre: server umount lustre-MDT0000 complete [13551.590464] LustreError: 435545:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781010386 with bad export cookie 1769198086080028833 [13551.593175] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13551.597616] LustreError: 435545:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13551.726144] Lustre: server umount lustre-MDT0001 complete [13555.746623] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [13556.964378] Key type lgssc unregistered [13557.144258] LNet: 438292:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13557.147796] LNetError: 438292:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13557.151596] LNet: Removed LNI 192.168.203.154@tcp [13558.176182] LNet: Removed LNI 192.168.203.154@tcp100 [13558.180228] LNetError: 438292:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13558.184387] LNetError: 438292:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [13558.188329] LNet: Removed LNI 192.168.203.154@tcp101 [13559.200165] LNet: Removed LNI 192.168.203.154@tcp102 [13559.202458] LNetError: 438292:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13559.205553] LNetError: 438292:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [13559.212298] LNet: Removed LNI 192.168.203.154@tcp103 [13560.224238] LNet: Removed LNI 192.168.203.154@tcp104 [13561.248856] LNet: Removed LNI 192.168.203.154@tcp105 [13561.250876] LNetError: 438292:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13561.255757] LNetError: 438292:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [13561.261668] LNet: Removed LNI 192.168.203.154@tcp106 [13562.272275] LNet: Removed LNI 192.168.203.154@tcp107 [13563.296209] LNet: Removed LNI 192.168.203.154@tcp108 [13564.320250] LNet: Removed LNI 192.168.203.154@tcp109 [13565.344184] LNet: Removed LNI 192.168.203.154@tcp110 [13565.346104] LNetError: 438292:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13565.349074] LNetError: 438292:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [13565.352385] LNet: Removed LNI 192.168.203.154@tcp111 [13566.371341] LNet: Removed LNI 192.168.203.154@tcp112 [13567.392370] LNet: Removed LNI 192.168.203.154@tcp113 [13568.416171] LNet: Removed LNI 192.168.203.154@tcp114 [13569.440119] LNet: Removed LNI 192.168.203.154@tcp115 [13570.464131] LNet: Removed LNI 192.168.203.154@tcp116 [13571.488156] LNet: Removed LNI 192.168.203.154@tcp117 [13572.512618] LNet: Removed LNI 192.168.203.154@tcp118 [13573.536236] LNet: Removed LNI 192.168.203.154@tcp119 [13573.538490] LNetError: 438292:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13573.541855] LNetError: 438292:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [13573.545120] LNet: Removed LNI 192.168.203.154@tcp120 [13574.560175] LNet: Removed LNI 192.168.203.154@tcp121 [13575.584558] LNet: Removed LNI 192.168.203.154@tcp122 [13576.608236] LNet: Removed LNI 192.168.203.154@tcp123 [13577.632157] LNet: Removed LNI 192.168.203.154@tcp124 [13577.634297] LNet: Removed LNI 192.168.203.154@tcp125 [13578.656291] LNet: Removed LNI 192.168.203.154@tcp126 [13579.680185] LNet: Removed LNI 192.168.203.154@tcp127 [13580.704244] LNet: Removed LNI 192.168.203.154@tcp128 [13581.728209] LNet: Removed LNI 192.168.203.154@tcp129 [13581.730459] LNet: Removed LNI 192.168.203.154@tcp130 [13581.732598] LNet: Removed LNI 192.168.203.154@tcp131 [13581.734395] LNet: Removed LNI 192.168.203.154@tcp132 [13581.736256] LNet: Removed LNI 192.168.203.154@tcp133 [13581.737889] LNet: Removed LNI 192.168.203.154@tcp134 [13581.745725] LNet: Removed LNI 192.168.203.154@tcp135 [13582.070316] Key type .llcrypt unregistered [13582.073655] Key type ._llcrypt unregistered [13590.531624] Key type ._llcrypt registered [13590.533013] Key type .llcrypt registered [13590.579208] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_hostid [13597.051788] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [13597.497623] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13597.530098] alg: No test for adler32 (adler32-zlib) [13598.408563] Lustre: Lustre: Build Version: 2.17.54_1_g695b805 [13598.507648] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [13600.096172] Key type lgssc registered [13600.512211] Lustre: Echo OBD driver; http://www.lustre.org/ [13613.452253] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 09:07:27 (1781010447) [13620.061276] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [13621.157419] Key type lgssc unregistered [13621.297353] LNet: 442302:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13621.299841] LNetError: 442302:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13621.309307] LNet: Removed LNI 192.168.203.154@tcp [13621.601898] Key type .llcrypt unregistered [13621.603836] Key type ._llcrypt unregistered [13628.785810] Key type ._llcrypt registered [13628.787365] Key type .llcrypt registered [13628.829506] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [13629.160810] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13629.236239] alg: No test for adler32 (adler32-zlib) [13630.093158] Lustre: Lustre: Build Version: 2.17.54_1_g695b805 [13630.178910] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [13631.760192] Key type lgssc registered [13632.149763] Lustre: Echo OBD driver; http://www.lustre.org/ [13635.572487] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13635.578339] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13636.670222] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13636.683251] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13636.721884] Lustre: lustre-MDT0000: new disk, initializing [13636.749244] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13636.757245] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13638.140335] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [13642.478545] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13642.521440] Lustre: 443475:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13642.546079] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13642.548588] Lustre: Skipped 1 previous similar message [13642.585455] Lustre: lustre-MDT0001: new disk, initializing [13642.622734] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13642.636428] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13642.641194] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13644.020251] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [13647.350518] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13648.692861] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13651.137232] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13651.243252] Lustre: lustre-OST0000: new disk, initializing [13651.246130] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13651.250048] Lustre: 445217:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13651.285989] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13653.393648] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [13656.694744] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13657.962615] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing lnet_if_list [13659.700425] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [13659.714595] LNet: Added LNI 192.168.203.154@tcp10 [8/256/0/180] [13660.160875] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13660.165302] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13660.211911] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13661.701546] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [13661.717319] LNet: Added LNI 192.168.203.154@tcp11 [8/256/0/180] [13663.015406] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [13663.028013] LNet: Added LNI 192.168.203.154@tcp12 [8/256/0/180] [13664.415626] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [13664.429585] LNet: Added LNI 192.168.203.154@tcp13 [8/256/0/180] [13665.949702] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [13665.964445] LNet: Added LNI 192.168.203.154@tcp14 [8/256/0/180] [13667.491429] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [13667.503951] LNet: Added LNI 192.168.203.154@tcp15 [8/256/0/180] [13669.035246] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [13669.049670] LNet: Added LNI 192.168.203.154@tcp16 [8/256/0/180] [13670.684616] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [13670.698788] LNet: Added LNI 192.168.203.154@tcp17 [8/256/0/180] [13672.373662] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [13672.386989] LNet: Added LNI 192.168.203.154@tcp18 [8/256/0/180] [13674.078163] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [13674.091982] LNet: Added LNI 192.168.203.154@tcp19 [8/256/0/180] [13675.737616] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [13675.750447] LNet: Added LNI 192.168.203.154@tcp20 [8/256/0/180] [13677.549850] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [13677.564093] LNet: Added LNI 192.168.203.154@tcp21 [8/256/0/180] [13679.402831] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [13679.415283] LNet: Added LNI 192.168.203.154@tcp22 [8/256/0/180] [13681.312866] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [13681.329771] LNet: Added LNI 192.168.203.154@tcp23 [8/256/0/180] [13683.271990] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [13683.285622] LNet: Added LNI 192.168.203.154@tcp24 [8/256/0/180] [13685.314517] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [13685.327559] LNet: Added LNI 192.168.203.154@tcp25 [8/256/0/180] [13687.357182] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [13687.371673] LNet: Added LNI 192.168.203.154@tcp26 [8/256/0/180] [13689.476082] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [13689.492112] LNet: Added LNI 192.168.203.154@tcp27 [8/256/0/180] [13691.522877] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [13691.536421] LNet: Added LNI 192.168.203.154@tcp28 [8/256/0/180] [13693.641287] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [13693.656694] LNet: Added LNI 192.168.203.154@tcp29 [8/256/0/180] [13695.784813] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [13695.799374] LNet: Added LNI 192.168.203.154@tcp30 [8/256/0/180] [13698.011424] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [13698.025282] LNet: Added LNI 192.168.203.154@tcp31 [8/256/0/180] [13700.140567] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [13700.154459] LNet: Added LNI 192.168.203.154@tcp32 [8/256/0/180] [13702.290270] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [13702.302642] LNet: Added LNI 192.168.203.154@tcp33 [8/256/0/180] [13704.484155] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [13704.498629] LNet: Added LNI 192.168.203.154@tcp34 [8/256/0/180] [13706.726149] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [13706.739340] LNet: Added LNI 192.168.203.154@tcp35 [8/256/0/180] [13709.029580] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [13709.041189] LNet: Added LNI 192.168.203.154@tcp36 [8/256/0/180] [13711.414809] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [13711.427701] LNet: Added LNI 192.168.203.154@tcp37 [8/256/0/180] [13713.791391] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [13713.804839] LNet: Added LNI 192.168.203.154@tcp38 [8/256/0/180] [13716.186629] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [13716.201793] LNet: Added LNI 192.168.203.154@tcp39 [8/256/0/180] [13718.741243] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [13718.753763] LNet: Added LNI 192.168.203.154@tcp40 [8/256/0/180] [13721.262750] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [13721.276700] LNet: Added LNI 192.168.203.154@tcp41 [8/256/0/180] [13723.697077] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [13723.709572] LNet: Added LNI 192.168.203.154@tcp42 [8/256/0/180] [13726.090589] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [13726.105112] LNet: Added LNI 192.168.203.154@tcp43 [8/256/0/180] [13728.777866] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [13728.793508] LNet: Added LNI 192.168.203.154@tcp44 [8/256/0/180] [13731.423604] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [13731.436973] LNet: Added LNI 192.168.203.154@tcp45 [8/256/0/180] [13734.141895] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [13734.156364] LNet: Added LNI 192.168.203.154@tcp46 [8/256/0/180] [13736.817267] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [13736.828826] LNet: Added LNI 192.168.203.154@tcp47 [8/256/0/180] [13739.594201] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [13739.606622] LNet: Added LNI 192.168.203.154@tcp48 [8/256/0/180] [13742.329499] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [13742.343439] LNet: Added LNI 192.168.203.154@tcp49 [8/256/0/180] [13745.410563] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [13745.424722] LNet: Added LNI 192.168.203.154@tcp50 [8/256/0/180] [13755.360848] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13755.364103] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13755.369426] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13757.409262] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13757.410810] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13759.655669] Lustre: server umount lustre-MDT0000 complete [13761.094089] LustreError: 443512:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781010595 with bad export cookie 7311111619469697937 [13761.097795] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13761.099029] LustreError: 443512:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13761.212364] Lustre: server umount lustre-MDT0001 complete [13768.907944] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [13772.261052] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13778.848164] Lustre: 442748:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781010597/real 1781010597] req@0000000040eebc3a x1867524829431040/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781010613 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13778.860058] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13778.871179] Lustre: Skipped 1 previous similar message [13782.688149] Lustre: 442749:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781010601/real 1781010601] req@000000009664edfd x1867524829431168/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781010617 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13784.032136] Lustre: 442751:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781010602/real 1781010602] req@000000009abcc216 x1867524829431296/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781010618 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13787.104145] Lustre: 442750:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781010606/real 1781010606] req@00000000d3c9af12 x1867524829431552/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781010622 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13787.104345] LustreError: 442747:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000003c6c1f0 x1867524829432192/t0(0) o250->MGC192.168.203.154@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13787.250965] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13788.747287] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [13792.353278] Lustre: 442748:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781010611/real 1781010611] req@00000000912cb866 x1867524829431680/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781010627 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13792.358513] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13792.564196] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13794.134024] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [13796.195425] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13797.610470] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13797.670367] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [13808.099832] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13808.100112] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13808.101456] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13808.101461] Lustre: Skipped 2 previous similar messages [13808.103271] LustreError: Skipped 1 previous similar message [13808.107580] Lustre: Skipped 1 previous similar message [13812.189802] Lustre: server umount lustre-OST0000 complete [13818.337695] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13818.338392] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13818.339033] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13818.339038] Lustre: Skipped 1 previous similar message [13818.351587] Lustre: Skipped 1 previous similar message [13819.739299] Lustre: server umount lustre-MDT0000 complete [13821.133740] LustreError: 448019:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781010656 with bad export cookie 7311111619469700016 [13821.138894] LustreError: 448019:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13821.139501] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13821.271308] Lustre: server umount lustre-MDT0001 complete [13825.190661] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [13826.247812] Key type lgssc unregistered [13826.421888] LNet: 471091:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13826.425777] LNetError: 471091:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13826.428768] LNet: Removed LNI 192.168.203.154@tcp [13826.431247] LNet: Removed LNI 192.168.203.154@tcp10 [13826.433127] LNet: Removed LNI 192.168.203.154@tcp11 [13826.435371] LNet: Removed LNI 192.168.203.154@tcp12 [13826.437367] LNet: Removed LNI 192.168.203.154@tcp13 [13826.439248] LNet: Removed LNI 192.168.203.154@tcp14 [13826.441145] LNet: Removed LNI 192.168.203.154@tcp15 [13826.442860] LNet: Removed LNI 192.168.203.154@tcp16 [13826.444450] LNet: Removed LNI 192.168.203.154@tcp17 [13826.446895] LNet: Removed LNI 192.168.203.154@tcp18 [13826.448941] LNet: Removed LNI 192.168.203.154@tcp19 [13826.450967] LNet: Removed LNI 192.168.203.154@tcp20 [13826.453210] LNet: Removed LNI 192.168.203.154@tcp21 [13826.455300] LNet: Removed LNI 192.168.203.154@tcp22 [13826.457246] LNet: Removed LNI 192.168.203.154@tcp23 [13826.458744] LNet: Removed LNI 192.168.203.154@tcp24 [13826.460240] LNet: Removed LNI 192.168.203.154@tcp25 [13826.461515] LNet: Removed LNI 192.168.203.154@tcp26 [13826.463042] LNet: Removed LNI 192.168.203.154@tcp27 [13826.464606] LNet: Removed LNI 192.168.203.154@tcp28 [13826.466188] LNet: Removed LNI 192.168.203.154@tcp29 [13826.467672] LNet: Removed LNI 192.168.203.154@tcp30 [13826.469149] LNet: Removed LNI 192.168.203.154@tcp31 [13826.470545] LNet: Removed LNI 192.168.203.154@tcp32 [13826.472192] LNet: Removed LNI 192.168.203.154@tcp33 [13826.473791] LNet: Removed LNI 192.168.203.154@tcp34 [13826.475757] LNet: Removed LNI 192.168.203.154@tcp35 [13826.477385] LNet: Removed LNI 192.168.203.154@tcp36 [13826.479320] LNet: Removed LNI 192.168.203.154@tcp37 [13826.481248] LNet: Removed LNI 192.168.203.154@tcp38 [13826.483223] LNet: Removed LNI 192.168.203.154@tcp39 [13826.484647] LNet: Removed LNI 192.168.203.154@tcp40 [13826.486063] LNet: Removed LNI 192.168.203.154@tcp41 [13826.487412] LNet: Removed LNI 192.168.203.154@tcp42 [13826.489212] LNet: Removed LNI 192.168.203.154@tcp43 [13826.490988] LNet: Removed LNI 192.168.203.154@tcp44 [13826.492624] LNet: Removed LNI 192.168.203.154@tcp45 [13826.494383] LNet: Removed LNI 192.168.203.154@tcp46 [13826.496108] LNet: Removed LNI 192.168.203.154@tcp47 [13826.497698] LNet: Removed LNI 192.168.203.154@tcp48 [13826.499081] LNet: Removed LNI 192.168.203.154@tcp49 [13826.505303] LNet: Removed LNI 192.168.203.154@tcp50 [13826.794940] Key type .llcrypt unregistered [13826.796720] Key type ._llcrypt unregistered [13834.175520] Key type ._llcrypt registered [13834.177142] Key type .llcrypt registered [13834.220976] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_hostid [13841.030638] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [13841.345898] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13841.354615] alg: No test for adler32 (adler32-zlib) [13842.218611] Lustre: Lustre: Build Version: 2.17.54_1_g695b805 [13842.308880] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [13843.896146] Key type lgssc registered [13844.378837] Lustre: Echo OBD driver; http://www.lustre.org/ [13857.585155] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13857.591544] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13858.698205] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13858.713169] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13858.752580] Lustre: lustre-MDT0000: new disk, initializing [13858.785239] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13858.793672] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13860.318371] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [13865.099404] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13865.151896] Lustre: 474225:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13865.182356] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13865.184791] Lustre: Skipped 1 previous similar message [13865.223483] Lustre: lustre-MDT0001: new disk, initializing [13865.262748] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13865.278705] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13865.282813] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13866.795552] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [13870.329128] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13871.749687] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13874.736460] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13874.822800] Lustre: *** cfs_fail_loc=90f, val=0*** [13874.860648] Lustre: lustre-OST0000: new disk, initializing [13874.862533] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13874.865734] Lustre: 476016:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13874.900649] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13877.099217] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [13880.581230] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13882.366223] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13882.370747] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13882.409060] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13887.458747] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13887.458849] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13887.462407] LustreError: Skipped 1 previous similar message [13887.463335] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13887.468073] Lustre: Skipped 1 previous similar message [13887.474373] Lustre: Skipped 1 previous similar message [13889.503768] Lustre: server umount lustre-OST0000 complete [13891.040866] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13891.043661] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13891.048817] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13892.577611] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13892.577912] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13897.697537] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13897.700257] Lustre: Skipped 2 previous similar messages [13902.818641] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13902.822312] Lustre: Skipped 1 previous similar message [13905.376117] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13905.448254] Lustre: server umount lustre-MDT0000 complete [13906.982384] LustreError: 474218:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781010741 with bad export cookie 10410257887140077048 [13906.986824] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13906.987108] LustreError: 474218:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [13907.121697] Lustre: server umount lustre-MDT0001 complete [13911.032809] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [13912.171543] Key type lgssc unregistered [13912.319574] LNet: 477788:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13912.322666] LNetError: 477788:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13912.330457] LNet: Removed LNI 192.168.203.154@tcp [13912.657393] Key type .llcrypt unregistered [13912.659187] Key type ._llcrypt unregistered [13920.629349] Key type ._llcrypt registered [13920.630668] Key type .llcrypt registered [13920.681968] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [13921.026267] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13921.092355] alg: No test for adler32 (adler32-zlib) [13921.958577] Lustre: Lustre: Build Version: 2.17.54_1_g695b805 [13922.053821] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [13923.640154] Key type lgssc registered [13924.063919] Lustre: Echo OBD driver; http://www.lustre.org/ [13927.083721] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13927.091162] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13928.269418] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13929.734698] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [13932.972127] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13933.114586] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13934.505409] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [13936.466967] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13937.819117] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13940.346485] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13940.464595] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13942.519937] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [13944.328365] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13958.920969] loop0: detected capacity change from 0 to 204800000 [13959.035353] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13959.040106] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13959.047124] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13961.162699] loop0: detected capacity change from 0 to 204800000 [13961.223886] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13961.229041] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13961.235454] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13963.057546] loop0: detected capacity change from 0 to 204800000 [13964.968807] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13965.017892] Lustre: fs15246-OST0000: new disk, initializing [13965.019786] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [13965.023291] Lustre: 482305:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13965.044236] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13966.997351] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [13970.978392] loop1: detected capacity change from 0 to 204800000 [13972.995582] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13973.044697] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [13973.073798] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [13973.112972] Lustre: fs15246-MDT0000: new disk, initializing [13973.181163] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13973.194226] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13973.236860] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [13974.305078] Lustre: 483411:0:(client.c:1638:after_reply()) @@@ resending request on EINPROGRESS req@000000008c446954 x1867525134516480/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1781010825 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [13974.630581] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [13983.392496] LustreError: 483411:0:(fid_request.c:213:seq_client_alloc_seq()) cli-cli-fs15246-OST0000-osc-MDT0000: Cannot allocate new meta-sequence: rc = -5 [13983.398895] LustreError: 483411:0:(fid_request.c:255:seq_client_get_seq()) cli-cli-fs15246-OST0000-osc-MDT0000: Can't allocate new sequence: rc = -5 [13983.404146] LustreError: 483411:0:(osp_precreate.c:460:osp_precreate_rollover_new_seq()) fs15246-OST0000-osc-MDT0000: alloc fid error: rc = -5 [13986.275653] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13986.280373] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [13988.960409] Lustre: server umount fs15246-MDT0000 complete [13991.400494] LustreError: 479026:0:(ldlm_lib.c:1179:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13991.407734] LustreError: 479026:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [13992.409261] Lustre: server umount fs15246-OST0000 complete [13996.000786] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13996.004074] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13996.009772] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14001.634262] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14001.636656] Lustre: Skipped 3 previous similar messages [14006.754615] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14006.756645] Lustre: Skipped 1 previous similar message [14010.336188] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14010.402056] Lustre: server umount lustre-OST0000 complete [14011.873830] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14011.874487] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14011.878681] Lustre: Skipped 2 previous similar messages [14011.881710] Lustre: Skipped 1 previous similar message [14017.823936] Lustre: server umount lustre-MDT0000 complete [14019.179616] LustreError: 479009:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781010854 with bad export cookie 6926720256285379948 [14019.182593] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14019.185581] LustreError: 479009:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [14019.310827] Lustre: server umount lustre-MDT0001 complete [14022.883209] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [14023.924736] Key type lgssc unregistered [14024.053406] LNet: 486470:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14024.056974] LNetError: 486470:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14024.063446] LNet: Removed LNI 192.168.203.154@tcp [14024.358971] Key type .llcrypt unregistered [14024.359937] Key type ._llcrypt unregistered [14031.884890] Key type ._llcrypt registered [14031.885966] Key type .llcrypt registered [14031.920715] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [14032.349392] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14032.356178] alg: No test for adler32 (adler32-zlib) [14033.214377] Lustre: Lustre: Build Version: 2.17.54_1_g695b805 [14033.308331] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [14034.896160] Key type lgssc registered [14035.301875] Lustre: Echo OBD driver; http://www.lustre.org/ [14048.342696] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_hostid [14051.874886] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [14064.161471] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14064.168538] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14065.272049] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14065.288841] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14065.333821] Lustre: lustre-MDT0000: new disk, initializing [14065.366386] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14065.375678] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14066.815635] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [14071.163804] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14071.213273] Lustre: 490853:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14071.244756] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14071.248238] Lustre: Skipped 1 previous similar message [14071.285926] Lustre: lustre-MDT0001: new disk, initializing [14071.325906] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14071.342327] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14071.347070] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14072.782814] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [14076.147134] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [14077.495668] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [14080.085205] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14080.205278] Lustre: lustre-OST0000: new disk, initializing [14080.207474] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14080.210749] Lustre: 492595:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14080.247904] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14082.367273] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14082.372060] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14082.414470] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14082.520278] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [14085.879658] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [14087.524700] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [14087.627834] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [14089.121531] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [14089.212939] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [14092.770674] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14092.775414] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14092.776755] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14092.781208] LustreError: Skipped 1 previous similar message [14092.783742] Lustre: Skipped 1 previous similar message [14096.017782] Lustre: server umount lustre-OST0000 complete [14097.891418] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14097.898323] Lustre: Skipped 1 previous similar message [14097.901350] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14097.905578] Lustre: Skipped 1 previous similar message [14103.009797] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14103.014268] Lustre: Skipped 1 previous similar message [14103.521366] Lustre: server umount lustre-MDT0000 complete [14104.861506] LustreError: 491632:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781010939 with bad export cookie 3668278020609596423 [14104.865174] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14104.866462] LustreError: 491632:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [14104.984811] Lustre: server umount lustre-MDT0001 complete [14113.656499] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_hostid [14117.162330] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [14128.470172] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 13866 sec ========= 09:16:02 (1781010962) [14129.103929] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 09:16:03 (1781010963) === [14130.342730] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 09:16:04 (1781010964) === [14144.260277] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [14145.331719] Key type lgssc unregistered [14145.466785] LNet: 499027:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14145.470406] LNetError: 499027:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14145.481313] LNet: Removed LNI 192.168.203.154@tcp [14145.797622] Key type .llcrypt unregistered [14145.799108] Key type ._llcrypt unregistered