[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 485603542 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2576MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001015] APIC: Switch to symmetric I/O mode setup [ 0.003356] x2apic enabled [ 0.004010] Switched APIC routing to physical x2apic. [ 0.006013] kvm-guest: setup PV IPIs [ 0.008985] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009018] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010011] pid_max: default: 32768 minimum: 301 [ 0.011130] LSM: Security Framework initializing [ 0.012050] Yama: becoming mindful. [ 0.013048] SELinux: Initializing. [ 0.014070] *** VALIDATE selinux *** [ 0.022750] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027606] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028141] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030025] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032022] *** VALIDATE tmpfs *** [ 0.033450] *** VALIDATE proc *** [ 0.035185] *** VALIDATE cgroup *** [ 0.036000] *** VALIDATE cgroup2 *** [ 0.037025] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038139] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040029] Spectre V2 : User space: Vulnerable [ 0.041009] Speculative Store Bypass: Vulnerable [ 0.044505] debug: unmapping init [mem 0xffffffffa1859000-0xffffffffa1860fff] [ 0.046188] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.047657] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.048036] ... version: 2 [ 0.049013] ... bit width: 48 [ 0.050011] ... generic registers: 4 [ 0.051011] ... value mask: 0000ffffffffffff [ 0.052011] ... max period: 00007fffffffffff [ 0.053012] ... fixed-purpose events: 3 [ 0.054014] ... event mask: 000000070000000f [ 0.055299] rcu: Hierarchical SRCU implementation. [ 0.057431] smp: Bringing up secondary CPUs ... [ 0.058604] x86: Booting SMP configuration: [ 0.059028] .... node #0, CPUs: #1 #2 #3 [ 0.062333] smp: Brought up 1 node, 4 CPUs [ 0.064024] smpboot: Max logical packages: 1 [ 0.065016] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.142783] node 0 deferred pages initialised in 74ms [ 0.145163] devtmpfs: initialized [ 0.146225] x86/mm: Memory block size: 128MB [ 0.149784] gcov: version magic: 0x41383552 [ 0.152292] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.157070] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.159351] pinctrl core: initialized pinctrl subsystem [ 0.162171] [ 0.162760] ************************************************************* [ 0.165013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.168013] ** ** [ 0.170011] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.172013] ** ** [ 0.175009] ** This means that this kernel is built to expose internal ** [ 0.177010] ** IOMMU data structures, which may compromise security on ** [ 0.179026] ** your system. ** [ 0.181012] ** ** [ 0.184011] ** If you see this message and you are not debugging the ** [ 0.186011] ** kernel, report this immediately to your vendor! ** [ 0.189014] ** ** [ 0.191009] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.192008] ************************************************************* [ 0.195711] NET: Registered protocol family 16 [ 0.197563] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.201070] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.204058] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.208014] cpuidle: using governor menu [ 0.209577] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.214063] PCI: Using configuration type 1 for base access [ 0.216123] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.227177] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.230030] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.234121] cryptd: max_cpu_qlen set to 1000 [ 0.238255] ACPI: Added _OSI(Module Device) [ 0.240016] ACPI: Added _OSI(Processor Device) [ 0.242014] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.244020] ACPI: Added _OSI(Processor Aggregator Device) [ 0.250006] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.257453] ACPI: Interpreter enabled [ 0.259063] ACPI: PM: (supports S0 S3 S4 S5) [ 0.260010] ACPI: Using IOAPIC for interrupt routing [ 0.261079] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.264374] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.273523] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.275037] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.278016] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.280067] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.285428] acpiphp: Slot [2] registered [ 0.287112] acpiphp: Slot [5] registered [ 0.289149] acpiphp: Slot [6] registered [ 0.290116] acpiphp: Slot [7] registered [ 0.292117] acpiphp: Slot [8] registered [ 0.294176] acpiphp: Slot [9] registered [ 0.295085] acpiphp: Slot [10] registered [ 0.297136] acpiphp: Slot [3] registered [ 0.298122] acpiphp: Slot [4] registered [ 0.299076] acpiphp: Slot [11] registered [ 0.300139] acpiphp: Slot [12] registered [ 0.301070] acpiphp: Slot [13] registered [ 0.303078] acpiphp: Slot [14] registered [ 0.304136] acpiphp: Slot [15] registered [ 0.306189] acpiphp: Slot [16] registered [ 0.307073] acpiphp: Slot [17] registered [ 0.309088] acpiphp: Slot [18] registered [ 0.310090] acpiphp: Slot [19] registered [ 0.312094] acpiphp: Slot [20] registered [ 0.313123] acpiphp: Slot [21] registered [ 0.314078] acpiphp: Slot [22] registered [ 0.316108] acpiphp: Slot [23] registered [ 0.318149] acpiphp: Slot [24] registered [ 0.320125] acpiphp: Slot [25] registered [ 0.322113] acpiphp: Slot [26] registered [ 0.323083] acpiphp: Slot [27] registered [ 0.324080] acpiphp: Slot [28] registered [ 0.326101] acpiphp: Slot [29] registered [ 0.327082] acpiphp: Slot [30] registered [ 0.329133] acpiphp: Slot [31] registered [ 0.330066] PCI host bridge to bus 0000:00 [ 0.332029] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.335124] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.338021] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.341024] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.345027] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.347022] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.349148] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.351993] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.355295] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.363563] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.368057] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.371015] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.373015] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.375019] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.378613] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.381844] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.384047] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.387760] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.393016] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.407017] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.412014] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.419354] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.427018] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.438014] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.458016] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.469564] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.479018] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.492015] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.517017] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.534595] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.543033] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.550024] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.567025] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.582443] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.604017] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.615016] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.642024] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.659718] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.669015] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.675017] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.700016] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.711847] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.719020] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.724016] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.746020] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.758306] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.760327] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.762335] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.764371] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.768285] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.773191] iommu: Default domain type: Passthrough [ 0.775490] SCSI subsystem initialized [ 0.777156] ACPI: bus type USB registered [ 0.779131] usbcore: registered new interface driver usbfs [ 0.781077] usbcore: registered new interface driver hub [ 0.782077] usbcore: registered new device driver usb [ 0.785174] pps_core: LinuxPPS API ver. 1 registered [ 0.787011] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.791063] PTP clock support registered [ 0.793076] EDAC MC: Ver: 3.0.0 [ 0.795051] PCI: Using ACPI for IRQ routing [ 0.796641] NetLabel: Initializing [ 0.798017] NetLabel: domain hash size = 128 [ 0.800012] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.802092] NetLabel: unlabeled traffic allowed by default [ 0.805139] vgaarb: loaded [ 0.806361] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.808020] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.815373] clocksource: Switched to clocksource kvm-clock [ 0.923592] VFS: Disk quotas dquot_6.6.0 [ 0.925616] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.928822] *** VALIDATE ramfs *** [ 0.930395] *** VALIDATE hugetlbfs *** [ 0.932658] pnp: PnP ACPI init [ 0.934861] pnp: PnP ACPI: found 6 devices [ 0.956875] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.959782] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.961778] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.963859] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.966126] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.968208] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.970699] NET: Registered protocol family 2 [ 0.973257] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.979678] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.982750] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.987727] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.990777] TCP: Hash tables configured (established 65536 bind 65536) [ 0.993423] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.996391] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.999156] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.001814] NET: Registered protocol family 1 [ 1.004277] RPC: Registered named UNIX socket transport module. [ 1.006203] RPC: Registered udp transport module. [ 1.007656] RPC: Registered tcp transport module. [ 1.009100] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.010971] NET: Registered protocol family 44 [ 1.012374] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.014162] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.015968] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.017883] PCI: CLS 0 bytes, default 64 [ 1.019271] Unpacking initramfs... [ 2.412691] debug: unmapping init [mem 0xffff9d5fbcc54000-0xffff9d5fbffbffff] [ 2.417138] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.419139] software IO TLB: mapped [mem 0x00000000b8c54000-0x00000000bcc54000] (64MB) [ 2.422073] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.915716] Initialise system trusted keyrings [ 2.917504] Key type blacklist registered [ 2.919564] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.930069] zbud: loaded [ 2.933953] *** VALIDATE nfs *** [ 2.935267] *** VALIDATE nfs4 *** [ 2.936713] pstore: using deflate compression [ 2.943196] Platform Keyring initialized [ 3.037621] NET: Registered protocol family 38 [ 3.039329] Key type asymmetric registered [ 3.040716] Asymmetric key parser 'x509' registered [ 3.042544] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.045860] io scheduler mq-deadline registered [ 3.047253] io scheduler kyber registered [ 3.048598] io scheduler bfq registered [ 3.050653] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.053521] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.056700] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.059261] ACPI: Power Button [PWRF] [ 3.064111] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.070397] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.087367] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.096118] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.110476] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.137070] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.167526] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.171947] Non-volatile memory driver v1.3 [ 3.173579] Linux agpgart interface v0.103 [ 3.202356] virtio_blk virtio1: [vda] 146696 512-byte logical blocks (75.1 MB/71.6 MiB) [ 3.205065] vda: detected capacity change from 0 to 75108352 [ 3.217650] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.220140] vdb: detected capacity change from 0 to 1073741824 [ 3.234103] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.238266] vdc: detected capacity change from 0 to 2621440000 [ 3.252288] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.255514] vdd: detected capacity change from 0 to 2621440000 [ 3.269415] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.272108] vde: detected capacity change from 0 to 4294967296 [ 3.287182] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.289220] vdf: detected capacity change from 0 to 4294967296 [ 3.295278] libphy: Fixed MDIO Bus: probed [ 3.302542] usbcore: registered new interface driver usbserial_generic [ 3.304925] usbserial: USB Serial support registered for generic [ 3.306935] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.309969] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.311773] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.314152] mousedev: PS/2 mouse device common for all mice [ 3.317243] rtc_cmos 00:05: RTC can wake from S4 [ 3.319625] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.321263] rtc_cmos 00:05: registered as rtc0 [ 3.326244] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.328140] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.329942] intel_pstate: CPU model not supported [ 3.336199] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.340693] hid: raw HID events driver (C) Jiri Kosina [ 3.342971] usbcore: registered new interface driver usbhid [ 3.345415] usbhid: USB HID core driver [ 3.347406] drop_monitor: Initializing network drop monitor service [ 3.349815] Initializing XFRM netlink socket [ 3.351733] NET: Registered protocol family 10 [ 3.356569] Segment Routing with IPv6 [ 3.358333] NET: Registered protocol family 17 [ 3.360631] mpls_gso: MPLS GSO support [ 3.366128] RAS: Correctable Errors collector initialized. [ 3.368329] AVX version of gcm_enc/dec engaged. [ 3.370390] AES CTR mode by8 optimization enabled [ 3.439502] sched_clock: Marking stable (3439467434, 0)->(4363569912, -924102478) [ 3.444171] registered taskstats version 1 [ 3.446854] Loading compiled-in X.509 certificates [ 3.449832] zswap: loaded using pool lzo/zbud [ 3.476831] Key type big_key registered [ 3.489656] Key type encrypted registered [ 3.491453] ima: No TPM chip found, activating TPM-bypass! [ 3.493499] ima: Allocated hash algorithm: sha1 [ 3.495171] ima: No architecture policies found [ 3.497283] evm: Initialising EVM extended attributes: [ 3.498988] evm: security.selinux [ 3.500379] evm: security.ima [ 3.501334] evm: security.capability [ 3.502887] evm: HMAC attrs: 0x1 [ 3.505214] rtc_cmos 00:05: setting system clock to 2026-09-08 05:26:23 UTC (1788845183) [ 3.512419] debug: unmapping init [mem 0xffffffffa2803000-0xffffffffa29fffff] [ 3.515522] debug: unmapping init [mem 0xffffffffa1582000-0xffffffffa1858fff] [ 3.523415] Write protecting the kernel read-only data: 28672k [ 3.526925] debug: unmapping init [mem 0xffffffff9fc03000-0xffffffff9fdfffff] [ 3.530152] debug: unmapping init [mem 0xffffffffa0514000-0xffffffffa05fffff] [ 3.562209] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.571389] systemd[1]: Detected virtualization kvm. [ 3.573432] systemd[1]: Detected architecture x86-64. [ 3.575629] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.602390] systemd[1]: No hostname configured. [ 3.604433] systemd[1]: Set hostname to . [ 3.606628] random: systemd: uninitialized urandom read (16 bytes read) [ 3.609328] systemd[1]: Initializing machine ID from random generator. [ 3.663492] random: ln: uninitialized urandom read (6 bytes read) [ 3.759162] random: systemd: uninitialized urandom read (16 bytes read) [ 3.761899] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.765957] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.769772] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ OK ] Reached target Timers. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket. Starting Create Volatile Files and Directories... [ OK ] Started Memstrack Anylazing Service. Starting Journal Service... Starting Apply Kernel Variables... Starting Setup Virtual Console... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.423618] device-mapper: uevent: version 1.0.3 [ 4.425941] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.095229] virtio_net virtio0 ens2: renamed from eth0 [ 5.122528] random: fast init done [ 5.142673] scsi host0: ata_piix [ 5.150211] scsi host1: ata_piix [ 5.151524] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.154086] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.801993] dracut-initqueue[578]: RTNETLINK answers: File exists [ 10.009650] random: crng init done [ 10.010754] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.356660] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.451880] printk: systemd: 25 output lines suppressed due to ratelimiting [ 11.687593] SELinux: Disabled at runtime. [ 11.755868] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.764258] systemd[1]: Detected virtualization kvm. [ 11.766020] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.211145] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.214842] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.220336] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.225514] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.229071] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.236207] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.241470] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target rpc_pipefs.target. Mounting POSIX Message Queue File System... [ OK ] Listening on udev Control Socket. [ OK ] Listening on udev Kernel Socket. Starting Remount Root and Kernel File Systems... Starting Create list of required st…ce nodes for the current kernel... Mounting Kernel Debug File System... Activating swap /dev/disk/by-label/SWAP... Starting udev Coldplug all Devices... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. Mounting Huge Pages File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK [ 12.357341] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Starting Apply Kernel Variables... [ OK ] Created slice system-sshd\x2dkeygen.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice system-getty.slice. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 12.699412] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.983680] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.002841] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.160288] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.179297] EDAC sbridge: Ver: 1.1.2 [ 14.663290] Key type dns_resolver registered [ 14.985128] NFS: Registering the id_resolver key type [ 14.987240] Key type id_resolver registered [ 14.989174] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... Starting Login Service... Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg304-server login: [ 65.454354] libcfs: loading out-of-tree module taints kernel. [ 65.534969] Key type ._llcrypt registered [ 65.538717] Key type .llcrypt registered [ 65.655541] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_hostid [ 83.714372] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 84.947099] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 84.982114] alg: No test for adler32 (adler32-zlib) [ 86.274947] Lustre: Lustre: Build Version: 2.17.57_108_g4a8ba26 [ 86.977542] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 88.752356] Key type lgssc registered [ 90.341760] Lustre: Echo OBD driver; http://www.lustre.org/ [ 99.438724] hrtimer: interrupt took 14273241 ns [ 105.291100] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 142.229340] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 154.101841] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 154.161541] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 155.410764] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 155.445155] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 155.553276] Lustre: lustre-MDT0000: new disk, initializing [ 155.632557] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 155.646883] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 160.735650] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 174.095284] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 174.226477] Lustre: 6502:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 174.318454] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 174.326155] Lustre: Skipped 1 previous similar message [ 174.419989] Lustre: lustre-MDT0001: new disk, initializing [ 174.482742] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 174.522052] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 174.533103] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 178.795156] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 183.653243] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 194.303262] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 194.500075] Lustre: lustre-OST0000: new disk, initializing [ 194.502349] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 194.505808] Lustre: 8438:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 194.553538] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 199.691890] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 199.697634] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 199.757923] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 201.156687] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 216.501147] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 216.654455] Lustre: lustre-OST0001: new disk, initializing [ 216.662470] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 216.671584] Lustre: 9512:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 216.743454] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 223.365840] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 225.853285] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 225.870465] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 225.954789] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 235.604751] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 242.456058] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 248.979388] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing check_logdir /tmp/testlogs/ [ 253.499937] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing yml_node [ 258.504866] Lustre: DEBUG MARKER: Client: 2.17.57.108 [ 260.726993] Lustre: DEBUG MARKER: MDS: 2.17.57.108 [ 262.683995] Lustre: DEBUG MARKER: OSS: 2.17.57.108 [ 264.186256] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Sep 8 01:30:42 EDT 2026 [ 279.658890] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 281.136513] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 297.441060] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 297.450124] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 297.460923] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 297.955209] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 297.967949] Lustre: Skipped 1 previous similar message [ 297.971893] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 297.977826] Lustre: Skipped 2 previous similar messages [ 303.078180] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 303.091326] Lustre: Skipped 3 previous similar messages [ 303.604203] Lustre: server umount lustre-MDT0000 complete [ 307.350735] LustreError: 6495:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788845487 with bad export cookie 13642715534678656794 [ 307.353397] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 307.375025] LustreError: 6495:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 308.193402] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 308.195652] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 308.234386] Lustre: Skipped 2 previous similar messages [ 308.247636] Lustre: Skipped 1 previous similar message [ 312.227888] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 312.248358] Lustre: Skipped 1 previous similar message [ 313.874767] Lustre: server umount lustre-MDT0001 complete [ 317.716690] Lustre: server umount lustre-OST0000 complete [ 321.725172] Lustre: server umount lustre-OST0001 complete [ 328.812172] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_hostid [ 339.017104] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 373.163160] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 373.448283] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 373.498398] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 373.606828] Lustre: lustre-MDT0000: new disk, initializing [ 373.717232] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 373.739697] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 378.227102] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 388.358264] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 388.480412] Lustre: 15062:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 388.539578] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 388.543220] Lustre: Skipped 1 previous similar message [ 388.636333] Lustre: lustre-MDT0001: new disk, initializing [ 388.779533] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 388.852113] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 388.863700] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 393.588943] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 402.381273] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 407.692866] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 414.149878] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 414.475201] Lustre: lustre-OST0000: new disk, initializing [ 414.480456] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 414.492298] Lustre: 17030:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 414.639166] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 415.759262] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 415.776254] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 415.924827] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 421.511820] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 430.410675] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 434.474214] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 434.781788] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 439.861190] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 440.185864] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 446.437138] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 446.458990] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 446.494340] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 457.184930] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 457.418103] Lustre: server umount lustre-OST0000 complete [ 462.306356] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 462.319943] Lustre: Skipped 2 previous similar messages [ 467.426862] LustreError: 15072:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 467.437728] LustreError: 15072:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 467.470730] Lustre: server umount lustre-MDT0000 complete [ 470.699924] LustreError: 15053:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788845650 with bad export cookie 13642715534678658334 [ 470.703617] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 470.716628] LustreError: 15053:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 471.204298] Lustre: server umount lustre-MDT0001 complete [ 476.649586] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 01:34:14 (1788845654) [ 479.186983] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 480.902228] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 01:34:19 (1788845659) [ 487.694617] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 488.223788] LustreError: 19442:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 488.336621] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 492.369086] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 493.540307] LustreError: 19443:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 498.663603] LustreError: 19442:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 500.518076] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 505.224675] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 511.956830] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 518.143383] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 525.215604] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 532.093985] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 540.086682] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 547.302719] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 547.309573] LustreError: Skipped 1 previous similar message [ 547.320161] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 547.333776] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 547.337410] Lustre: Skipped 7 previous similar messages [ 549.353097] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 553.488598] Lustre: server umount lustre-OST0000 complete [ 557.029843] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 571.360399] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 571.592651] Lustre: server umount lustre-MDT0000 complete [ 574.949165] LustreError: 19442:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 574.976300] LustreError: 19442:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 574.994664] LustreError: 20177:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788845754 with bad export cookie 13642715534678659216 [ 575.004214] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 575.012089] LustreError: 20177:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 575.324188] Lustre: server umount lustre-MDT0001 complete [ 585.601891] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 588.361458] Key type lgssc unregistered [ 588.679537] LNet: 22974:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 588.692491] LNetError: 22974:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 588.710995] LNet: Removed LNI 192.168.203.104@tcp [ 589.617262] Key type .llcrypt unregistered [ 589.619461] Key type ._llcrypt unregistered [ 610.928821] Key type ._llcrypt registered [ 610.938575] Key type .llcrypt registered [ 611.079282] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 612.297161] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 612.312078] alg: No test for adler32 (adler32-zlib) [ 613.426868] Lustre: Lustre: Build Version: 2.17.57_108_g4a8ba26 [ 613.781413] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 615.536171] Key type lgssc registered [ 616.741921] Lustre: Echo OBD driver; http://www.lustre.org/ [ 625.049746] loop: module loaded [ 625.161232] loop0: detected capacity change from 0 to 204800000 [ 625.395514] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 625.407114] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 625.867857] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 631.916992] loop0: detected capacity change from 0 to 204800000 [ 632.182068] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 632.195663] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 632.215309] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 636.973344] loop0: detected capacity change from 0 to 204800000 [ 642.052901] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 642.077411] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 643.282388] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 643.344796] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 643.412953] Lustre: lustre-MDT0000: new disk, initializing [ 643.496355] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 643.517129] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 647.347311] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 655.659074] loop1: detected capacity change from 0 to 204800000 [ 661.226734] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 661.555162] Lustre: lustre-OST07c6: new disk, initializing [ 661.561205] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 661.567366] Lustre: Skipped 1 previous similar message [ 661.574938] Lustre: 26152:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 661.665257] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 666.907232] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 668.253626] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 668.260651] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 668.301506] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 670.906671] Lustre: 26517:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 670.914306] Lustre: 26517:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 670.927607] Lustre: 26517:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 670.934991] Lustre: 26517:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 670.947628] Lustre: 26517:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 670.955657] Lustre: 26517:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 688.584255] Lustre: 25140:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 688.592986] Lustre: 25140:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 688.600853] Lustre: 25140:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 688.617709] Lustre: 25140:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 688.630202] Lustre: 25140:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 688.644943] Lustre: 25140:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 693.591928] Lustre: 26517:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 693.599917] Lustre: 26517:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 693.611824] Lustre: 26517:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 693.620297] Lustre: 26517:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 693.630414] Lustre: 26517:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 693.637689] Lustre: 26517:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 693.647988] Lustre: 26517:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 693.656422] Lustre: 26517:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 693.663317] Lustre: 26517:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 693.675338] Lustre: 26517:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 693.681651] Lustre: 26517:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 693.689871] Lustre: 26517:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 696.891240] Lustre: 25139:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 696.920677] Lustre: 25139:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 696.933295] Lustre: 25139:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 696.941526] Lustre: 25139:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 696.953601] Lustre: 25139:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 696.961109] Lustre: 25139:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 696.967697] Lustre: 25139:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/0 [ 696.980240] Lustre: 25139:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 696.985487] Lustre: 25139:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 696.995051] Lustre: 25139:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 697.003612] Lustre: 25139:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 697.010191] Lustre: 25139:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 701.702883] Lustre: 26517:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 701.713411] Lustre: 26517:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 701.727958] Lustre: 26517:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 701.737677] Lustre: 26517:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 701.752962] Lustre: 26517:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 701.772390] Lustre: 26517:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 701.784964] Lustre: 26517:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 701.801701] Lustre: 26517:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 701.810991] Lustre: 26517:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 701.819443] Lustre: 26517:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 701.824104] Lustre: 26517:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 701.828162] Lustre: 26517:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 712.592632] Lustre: 25139:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 712.598874] Lustre: 25139:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 5 previous similar messages [ 712.603109] Lustre: 25139:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 712.608104] Lustre: 25139:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 712.613366] Lustre: 25139:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 712.618464] Lustre: 25139:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 712.623280] Lustre: 25139:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 712.628650] Lustre: 25139:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 712.633445] Lustre: 25139:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 712.637398] Lustre: 25139:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 712.641469] Lustre: 25139:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 712.646120] Lustre: 25139:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 758.731667] Lustre: server umount lustre-OST07c6 complete [ 776.288422] Lustre: server umount lustre-MDT0000 complete [ 790.911964] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 793.208571] Key type lgssc unregistered [ 793.438556] LNet: 28712:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 793.443218] LNetError: 28712:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 793.456193] LNet: Removed LNI 192.168.203.104@tcp [ 794.069953] Key type .llcrypt unregistered [ 794.072680] Key type ._llcrypt unregistered [ 810.321932] Key type ._llcrypt registered [ 810.325235] Key type .llcrypt registered [ 810.404968] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 811.507116] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 811.573096] alg: No test for adler32 (adler32-zlib) [ 812.561798] Lustre: Lustre: Build Version: 2.17.57_108_g4a8ba26 [ 812.799528] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 814.440262] Key type lgssc registered [ 815.590907] Lustre: Echo OBD driver; http://www.lustre.org/ [ 822.437460] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 822.457631] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 823.977783] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 828.454832] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 833.507849] LustreError: 29978:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 833.520590] LustreError: 29978:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 836.936989] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 837.252403] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 841.645789] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 848.589738] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 854.173820] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 860.201909] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 860.617884] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 866.869383] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 868.811426] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 874.859691] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 889.319296] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 889.328749] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 889.336245] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 889.346918] Lustre: Skipped 1 previous similar message [ 889.375834] Lustre: Skipped 1 previous similar message [ 894.446569] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 894.455757] Lustre: Skipped 1 previous similar message [ 895.375574] Lustre: server umount lustre-OST0000 complete [ 899.554258] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 899.571890] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 899.582525] Lustre: Skipped 1 previous similar message [ 899.602194] Lustre: Skipped 1 previous similar message [ 904.673805] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 904.681530] Lustre: Skipped 1 previous similar message [ 905.995882] Lustre: server umount lustre-MDT0000 complete [ 909.536300] LustreError: 31819:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788846089 with bad export cookie 6077264103578157429 [ 909.540798] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 909.544658] LustreError: 31819:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 909.810215] Lustre: server umount lustre-MDT0001 complete [ 920.738666] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 924.421560] Key type lgssc unregistered [ 924.805434] LNet: 33711:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 924.849392] LNetError: 33711:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 925.934533] LNet: Removed LNI 192.168.203.104@tcp [ 926.987612] Key type .llcrypt unregistered [ 926.990306] Key type ._llcrypt unregistered [ 945.531558] Key type ._llcrypt registered [ 945.535618] Key type .llcrypt registered [ 945.646421] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 946.779254] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 946.847642] alg: No test for adler32 (adler32-zlib) [ 947.975654] Lustre: Lustre: Build Version: 2.17.57_108_g4a8ba26 [ 948.211672] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 949.928195] Key type lgssc registered [ 950.992071] Lustre: Echo OBD driver; http://www.lustre.org/ [ 958.372272] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 958.403376] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 959.810679] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 964.577739] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 969.201350] LustreError: 34979:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 969.229395] LustreError: 34979:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 973.678258] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 974.077985] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 978.572420] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 986.417364] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 991.794840] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 998.164038] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 998.544372] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1004.082666] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1004.740508] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1010.979096] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1025.511058] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1025.515271] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1025.516102] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1025.522361] Lustre: Skipped 1 previous similar message [ 1025.537946] LustreError: Skipped 1 previous similar message [ 1026.871971] Lustre: server umount lustre-OST0000 complete [ 1030.628823] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1030.639341] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1030.642290] Lustre: Skipped 1 previous similar message [ 1035.746764] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1035.757245] Lustre: Skipped 1 previous similar message [ 1036.483551] Lustre: server umount lustre-MDT0000 complete [ 1040.290935] LustreError: 36822:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788846220 with bad export cookie 13281400149954274106 [ 1040.298065] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1040.310591] LustreError: 36822:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1040.634096] Lustre: server umount lustre-MDT0001 complete [ 1051.416347] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 1054.330902] Key type lgssc unregistered [ 1054.700192] LNet: 38509:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1054.712468] LNetError: 38509:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1054.733045] LNet: Removed LNI 192.168.203.104@tcp [ 1055.724065] Key type .llcrypt unregistered [ 1055.725840] Key type ._llcrypt unregistered [ 1079.751482] Key type ._llcrypt registered [ 1079.752981] Key type .llcrypt registered [ 1079.900361] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 1080.732483] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1080.775047] alg: No test for adler32 (adler32-zlib) [ 1081.894421] Lustre: Lustre: Build Version: 2.17.57_108_g4a8ba26 [ 1082.155937] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 1083.912283] Key type lgssc registered [ 1085.248967] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1094.283739] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1094.294715] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1095.547415] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1100.766492] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1105.893409] LustreError: 39923:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1105.926268] LustreError: 39923:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1110.322333] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1110.552676] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1115.156637] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1122.876682] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1128.467833] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1135.439850] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1135.799534] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1140.924603] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1142.618401] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1151.061952] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1161.697628] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1161.712966] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1161.731542] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1164.048366] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.4@tcp (stopping) [ 1165.799907] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1165.802239] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1165.819519] Lustre: Skipped 2 previous similar messages [ 1167.526627] Lustre: server umount lustre-MDT0000 complete [ 1169.129942] LustreError: 41318:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.4@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1170.917535] LustreError: 39920:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1170.941729] LustreError: 39920:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1171.261718] LustreError: 39905:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788846351 with bad export cookie 5383937934385459036 [ 1171.278731] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1174.250048] LustreError: 39919:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.4@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1174.289510] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.4@tcp (stopping) [ 1176.052933] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1176.083034] Lustre: Skipped 1 previous similar message [ 1177.716549] Lustre: server umount lustre-MDT0001 complete [ 1192.719499] Lustre: server umount lustre-OST0000 complete [ 1210.257877] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 1213.530296] Key type lgssc unregistered [ 1213.801041] LNet: 43782:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1213.811520] LNetError: 43782:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1213.837307] LNet: Removed LNI 192.168.203.104@tcp [ 1214.640506] Key type .llcrypt unregistered [ 1214.643121] Key type ._llcrypt unregistered [ 1246.877292] Key type ._llcrypt registered [ 1246.879072] Key type .llcrypt registered [ 1246.997074] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 1248.176557] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1248.203820] alg: No test for adler32 (adler32-zlib) [ 1249.275466] Lustre: Lustre: Build Version: 2.17.57_108_g4a8ba26 [ 1249.598449] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 1251.256171] Key type lgssc registered [ 1252.364709] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1260.588517] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1260.621492] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1262.398481] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1267.363991] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1272.811774] LustreError: 45197:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1272.823492] LustreError: 45197:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1276.098535] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1276.464475] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1280.778931] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1288.898426] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1294.055924] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1300.619257] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1301.056368] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1307.178934] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1307.819296] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1315.533390] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1322.469426] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1322.473443] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1322.479855] LustreError: Skipped 1 previous similar message [ 1322.481101] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1322.501069] Lustre: Skipped 1 previous similar message [ 1323.278792] Lustre: lustre-OST0000: Not available for connect from 192.168.203.4@tcp (stopping) [ 1323.290491] Lustre: Skipped 1 previous similar message [ 1327.599449] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1327.613735] Lustre: Skipped 1 previous similar message [ 1328.512618] Lustre: server umount lustre-OST0000 complete [ 1348.066335] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1348.069071] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1348.082027] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1348.082036] Lustre: Skipped 1 previous similar message [ 1352.162124] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1352.172535] Lustre: Skipped 2 previous similar messages [ 1360.352139] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1360.598750] Lustre: server umount lustre-MDT0000 complete [ 1362.411417] LustreError: 45197:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1362.435257] LustreError: 45197:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1364.687767] LustreError: 45177:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788846544 with bad export cookie 11022800003027956485 [ 1364.701277] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1364.702583] LustreError: 45177:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1379.296334] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1379.588253] Lustre: server umount lustre-MDT0001 complete [ 1389.878735] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 1392.987802] Key type lgssc unregistered [ 1393.215573] LNet: 48910:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1393.223285] LNetError: 48910:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1393.233169] LNet: Removed LNI 192.168.203.104@tcp [ 1394.145294] Key type .llcrypt unregistered [ 1394.148771] Key type ._llcrypt unregistered [ 1419.400152] Key type ._llcrypt registered [ 1419.401920] Key type .llcrypt registered [ 1419.545551] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 1420.866646] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1420.927331] alg: No test for adler32 (adler32-zlib) [ 1421.926779] Lustre: Lustre: Build Version: 2.17.57_108_g4a8ba26 [ 1422.198681] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 1423.896205] Key type lgssc registered [ 1424.913967] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1432.708057] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1432.736445] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1434.415437] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1439.510547] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1444.838517] LustreError: 50330:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1444.856611] LustreError: 50330:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1448.708668] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1449.187334] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1454.034897] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1461.706235] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1467.768754] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1473.778357] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1474.034309] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1479.172521] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1480.289730] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1488.261822] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1493.435927] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1497.372492] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1510.062762] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-09-08 1h51m27s [ 1511.740723] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1514.978630] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1514.980447] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1515.018052] Lustre: Skipped 1 previous similar message [ 1515.775883] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.4@tcp (stopping) [ 1515.783536] Lustre: Skipped 2 previous similar messages [ 1516.842232] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.4@tcp (stopping) [ 1516.849222] Lustre: Skipped 3 previous similar messages [ 1521.897644] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.4@tcp (stopping) [ 1528.037985] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.4@tcp (stopping) [ 1528.060389] Lustre: Skipped 3 previous similar messages [ 1528.288406] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1528.531632] Lustre: server umount lustre-MDT0000 complete [ 1532.011144] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1534.189597] LustreError: 50324:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.4@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1538.937420] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1539.074884] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1539.373952] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1539.482630] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1540.394159] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1543.890192] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1547.207876] Lustre: DEBUG MARKER: Wait for df (36699) ... [ 1548.755511] Lustre: DEBUG MARKER: done [ 1554.913599] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1554.926340] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1554.957805] Lustre: Skipped 1 previous similar message [ 1554.984497] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1554.992336] Lustre: Skipped 3 previous similar messages [ 1555.949107] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1559.628892] Lustre: server umount lustre-OST0000 complete [ 1566.178256] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1566.179918] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1566.208631] Lustre: Skipped 1 previous similar message [ 1569.751530] Lustre: server umount lustre-MDT0000 complete [ 1573.351644] LustreError: 52169:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1573.386720] LustreError: 52169:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 1573.723110] LustreError: 50310:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788846753 with bad export cookie 12260583057304041366 [ 1573.736582] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1574.399534] Lustre: server umount lustre-MDT0001 complete [ 1586.989866] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 1590.207352] Key type lgssc unregistered [ 1590.515196] LNet: 55251:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1590.534711] LNetError: 55251:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1590.557162] LNet: Removed LNI 192.168.203.104@tcp [ 1591.453092] Key type .llcrypt unregistered [ 1591.461272] Key type ._llcrypt unregistered [ 1633.027938] Key type ._llcrypt registered [ 1633.030920] Key type .llcrypt registered [ 1633.143320] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 1634.248335] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1634.464911] alg: No test for adler32 (adler32-zlib) [ 1635.591813] Lustre: Lustre: Build Version: 2.17.57_108_g4a8ba26 [ 1635.934770] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 1637.616188] Key type lgssc registered [ 1638.887110] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1646.959676] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1647.020099] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1648.453061] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1648.482410] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1648.638305] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1652.622849] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1661.100879] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1661.270364] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1661.336477] Lustre: 57458:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1661.702646] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1666.552948] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1674.912148] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1681.140959] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1687.930321] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1688.152848] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1688.378928] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1694.876473] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1696.460975] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1702.101976] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1707.868310] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-09-08 1h54m46s [ 1709.383426] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1715.796138] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1718.226349] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1718.260719] LustreError: 57463:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1738.523484] Lustre: lustre-MDT0000: Client bcb17a64-7cd5-4417-8189-104f570f4d3d (at 192.168.203.4@tcp) reconnecting [ 1758.264114] LustreError: 57463:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1760.646702] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1763.179896] Lustre: DEBUG MARKER: done [ 1773.538708] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1773.540526] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1773.550406] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1773.560014] Lustre: Skipped 1 previous similar message [ 1773.576541] Lustre: Skipped 1 previous similar message [ 1775.785432] Lustre: server umount lustre-OST0000 complete [ 1783.779259] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1783.785868] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1783.806946] Lustre: Skipped 1 previous similar message [ 1783.851234] Lustre: Skipped 1 previous similar message [ 1784.940772] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1784.946613] Lustre: Skipped 1 previous similar message [ 1786.562854] Lustre: server umount lustre-MDT0000 complete [ 1790.110973] LustreError: 60219:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788846970 with bad export cookie 17708394027383770023 [ 1790.125030] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1790.128726] LustreError: 60219:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1790.505501] Lustre: server umount lustre-MDT0001 complete [ 1801.862691] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 1804.836459] Key type lgssc unregistered [ 1805.194710] LNet: 61793:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1805.199932] LNetError: 61793:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1805.221845] LNet: Removed LNI 192.168.203.104@tcp [ 1806.191310] Key type .llcrypt unregistered [ 1806.194116] Key type ._llcrypt unregistered [ 1849.129230] Key type ._llcrypt registered [ 1849.133897] Key type .llcrypt registered [ 1849.264291] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 1850.379311] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1850.444063] alg: No test for adler32 (adler32-zlib) [ 1851.458485] Lustre: Lustre: Build Version: 2.17.57_108_g4a8ba26 [ 1851.751439] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 1853.440222] Key type lgssc registered [ 1854.687811] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1857.956504] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1857.985782] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1859.263352] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1859.277186] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1859.440231] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1860.579774] Lustre: Failing over lustre-MDT0000 [ 1861.211564] Lustre: server umount lustre-MDT0000 complete [ 1869.222258] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 01:57:27 (1788847047) [ 1876.442379] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1877.067187] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1882.363932] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1890.364215] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1890.439767] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1890.460651] Lustre: 64587:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1890.768468] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1894.925873] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1901.803693] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1907.117695] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1913.556810] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1913.766450] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1914.078261] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1920.214868] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1920.922653] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1928.829484] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1933.787665] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1940.905633] Lustre: server umount lustre-MDT0000 complete [ 1940.965591] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1940.978974] Lustre: Skipped 2 previous similar messages [ 1944.456131] LustreError: 66441:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788847124 with bad export cookie 17881182719890880587 [ 1944.468741] LustreError: 66441:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1944.470135] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1944.715334] Lustre: server umount lustre-MDT0001 complete [ 1947.864625] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1952.366105] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1960.234819] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1962.208174] Lustre: 63510:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788847126/real 1788847126] req@0000000050e315f8 x1875742064416640/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788847142 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1962.226622] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1965.537956] Lustre: 63507:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788847129/real 1788847129] req@00000000a2308f7d x1875742064416768/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788847145 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1967.648292] Lustre: 63507:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788847131/real 1788847131] req@00000000a731d7bc x1875742064416896/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788847147 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1969.441046] LustreError: 63506:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000001718b6d9 x1875742064417792/t0(0) o250->MGC192.168.203.104@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1969.880591] LustreError: 66433:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1969.907668] LustreError: 66433:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 1970.029863] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1970.155525] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1970.976460] Lustre: 63509:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788847134/real 1788847134] req@000000006bbe2f82 x1875742064417152/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788847150 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1972.083351] LustreError: 68358:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1972.097509] LustreError: 68358:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1974.873489] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1975.138610] Lustre: 63509:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788847139/real 1788847139] req@000000000fd2c225 x1875742064417280/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788847155 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1977.318503] LustreError: 68356:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1977.338384] LustreError: 68356:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1981.415529] LustreError: 68477:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1981.418031] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1981.436316] LustreError: 68477:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1984.231698] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1984.603976] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1989.004828] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1989.614401] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1997.449303] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2003.295339] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2010.097219] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2010.106947] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2010.112873] Lustre: Skipped 2 previous similar messages [ 2010.118141] Lustre: Skipped 1 previous similar message [ 2010.766322] Lustre: server umount lustre-MDT0000 complete [ 2015.211367] LustreError: 68477:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2015.248820] LustreError: 68477:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2015.467486] LustreError: 64579:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788847195 with bad export cookie 17881182719890882862 [ 2015.469780] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2015.492241] LustreError: 64579:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2015.857903] Lustre: server umount lustre-MDT0001 complete [ 2022.720883] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2031.625181] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2036.705155] Lustre: 63509:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788847200/real 1788847200] req@000000007040f93a x1875742064444288/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788847216 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2036.749331] Lustre: 63509:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2036.780050] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2040.933552] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xf826b8f1f6f30691 [ 2040.957564] Lustre: MGC192.168.203.104@tcp: Connection restored to 0@lo (at 0@lo) [ 2041.254261] LustreError: 66434:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2041.338537] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2041.405995] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2046.318972] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2054.794920] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2058.925751] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2060.274187] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2065.391508] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2067.057875] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2073.763501] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2085.864392] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2085.865523] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2085.871807] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2085.884864] Lustre: Skipped 2 previous similar messages [ 2085.920755] Lustre: Skipped 3 previous similar messages [ 2087.994368] Lustre: server umount lustre-MDT0000 complete [ 2090.995939] LustreError: 71040:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2091.011226] LustreError: 71040:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 2091.768681] LustreError: 64579:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788847271 with bad export cookie 17881182719890884241 [ 2091.779050] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2092.097512] Lustre: server umount lustre-MDT0001 complete [ 2098.012869] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2100.797678] Lustre: DEBUG MARKER: files compared the same [ 2112.949196] Lustre: server umount lustre-OST0000 complete [ 2128.966755] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 2131.855289] Key type lgssc unregistered [ 2132.222574] LNet: 74332:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2132.229359] LNetError: 74332:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2132.256927] LNet: Removed LNI 192.168.203.104@tcp [ 2133.111980] Key type .llcrypt unregistered [ 2133.114551] Key type ._llcrypt unregistered [ 2155.656664] Key type ._llcrypt registered [ 2155.659807] Key type .llcrypt registered [ 2155.782160] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 2157.251934] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2157.326474] alg: No test for adler32 (adler32-zlib) [ 2158.384508] Lustre: Lustre: Build Version: 2.17.57_108_g4a8ba26 [ 2158.587548] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 2160.312204] Key type lgssc registered [ 2161.223707] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2169.168196] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2169.189923] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2170.800602] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2175.252672] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug +malloc all [ 2180.077492] LustreError: 75745:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2180.096339] LustreError: 75745:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2182.702281] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2183.026722] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2187.969194] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug +malloc all [ 2196.067208] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2202.070993] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2208.372267] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2208.634034] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2215.523400] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug +malloc all [ 2215.814078] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2223.438840] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2236.385706] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2236.399624] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2236.411700] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2237.412280] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2237.415685] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2237.440408] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2240.440080] Lustre: server umount lustre-OST0000 complete [ 2244.577043] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2244.587370] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2244.610769] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2244.620762] Lustre: Skipped 1 previous similar message [ 2247.648699] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2247.652993] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2252.772739] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2252.780977] Lustre: Skipped 2 previous similar messages [ 2258.400199] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2258.547746] Lustre: server umount lustre-MDT0000 complete [ 2262.271421] LustreError: 75725:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788847442 with bad export cookie 7429773279118273379 [ 2262.273148] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2262.283689] LustreError: 75725:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2262.664396] Lustre: server umount lustre-MDT0001 complete [ 2272.775552] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 2275.573411] Key type lgssc unregistered [ 2275.902621] LNet: 79278:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2275.914218] LNetError: 79278:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2276.971572] LNet: Removed LNI 192.168.203.104@tcp [ 2277.978543] Key type .llcrypt unregistered [ 2277.982359] Key type ._llcrypt unregistered [ 2300.575524] Key type ._llcrypt registered [ 2300.577454] Key type .llcrypt registered [ 2300.665798] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 2302.107981] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2302.156475] alg: No test for adler32 (adler32-zlib) [ 2303.216642] Lustre: Lustre: Build Version: 2.17.57_108_g4a8ba26 [ 2303.495718] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 2305.176642] Key type lgssc registered [ 2306.662098] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2314.825200] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2314.868647] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2330.400891] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2335.520771] LustreError: 80665:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.104@tcp: failed processing log, type 4: rc = -110 [ 2362.208203] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2368.386721] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2375.794729] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2383.057816] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2383.269344] Lustre: *** cfs_fail_loc=706, val=0*** [ 2383.696158] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2383.809847] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2388.512283] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2388.964190] LustreError: 80691:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2388.990623] LustreError: 80691:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2394.082732] LustreError: 80690:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2396.150581] LustreError: 81754:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2396.161943] LustreError: 81754:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2396.883336] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2397.223562] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2401.724586] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2408.459617] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2414.108903] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2417.634019] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2417.646207] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2417.652308] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2417.661192] Lustre: Skipped 1 previous similar message [ 2417.672987] Lustre: Skipped 1 previous similar message [ 2422.383836] Lustre: server umount lustre-OST0000 complete [ 2427.874421] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2427.874571] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2427.906618] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2432.460591] Lustre: server umount lustre-MDT0000 complete [ 2433.001212] LustreError: 81758:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2433.013478] LustreError: 81758:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2435.972194] LustreError: 80672:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788847615 with bad export cookie 17273308754969301732 [ 2435.990620] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2435.996064] LustreError: 80672:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2436.426877] Lustre: server umount lustre-MDT0001 complete [ 2447.287314] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 2449.958381] Key type lgssc unregistered [ 2450.237716] LNet: 84281:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2450.246816] LNetError: 84281:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2450.263957] LNet: Removed LNI 192.168.203.104@tcp [ 2451.008337] Key type .llcrypt unregistered [ 2451.010081] Key type ._llcrypt unregistered [ 2474.874951] Key type ._llcrypt registered [ 2474.878854] Key type .llcrypt registered [ 2475.000828] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 2475.951431] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2476.117326] alg: No test for adler32 (adler32-zlib) [ 2477.217635] Lustre: Lustre: Build Version: 2.17.57_108_g4a8ba26 [ 2477.489372] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 2479.208172] Key type lgssc registered [ 2480.809354] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2488.327353] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2488.367985] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2493.667479] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2500.530842] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2501.121167] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2505.743549] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2506.211846] LustreError: 86381:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2506.229181] LustreError: 86381:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2511.339956] LustreError: 86379:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2511.359352] LustreError: 86379:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2513.885444] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2514.320790] LustreError: 86384:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2514.345552] LustreError: 86384:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2514.382451] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2519.541221] LustreError: 86385:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2519.575556] LustreError: 86385:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2521.366860] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2524.652256] LustreError: 86384:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2524.668051] LustreError: 86384:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2530.329643] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2530.358332] Lustre: MGS: Client a81951e2-4d5a-46a7-a6f3-49d10db47c84 (at 0@lo) reconnecting [ 2530.405810] LustreError: 86395:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2530.684664] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2530.789337] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2535.356328] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2551.267416] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2551.284491] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2551.292679] Lustre: Skipped 1 previous similar message [ 2551.305326] Lustre: Skipped 1 previous similar message [ 2556.227311] Lustre: server umount lustre-OST0000 complete [ 2561.505312] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2561.512336] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2561.527807] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2561.546599] Lustre: Skipped 1 previous similar message [ 2565.950781] Lustre: server umount lustre-MDT0000 complete [ 2566.627315] LustreError: 86384:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2566.647226] LustreError: 86384:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2569.867731] Lustre: server umount lustre-MDT0001 complete [ 2573.274937] Lustre: server umount MGS complete [ 2583.692348] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 02:09:22 (1788847762) [ 2616.767376] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_hostid [ 2626.650486] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 2662.690030] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2667.455397] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2676.392137] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2676.597053] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2676.668523] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2676.740888] Lustre: lustre-MDT0001: new disk, initializing [ 2676.843518] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2676.890716] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2676.904964] Lustre: 93573:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2676.927225] LustreError: 93573:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2680.973961] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2690.382203] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2690.780527] Lustre: lustre-OST0000: new disk, initializing [ 2690.784950] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2690.805465] Lustre: 94338:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2690.893787] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2699.943361] Lustre: 94627:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@000000005bc65710 x1875742720827136/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788847895 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2701.418920] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2713.242599] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2719.762507] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2719.770556] Lustre: MGS: Client 5f3eb053-ad1a-4fa4-9286-6b6c7abc3261 (at 0@lo) reconnecting [ 2719.804353] Lustre: 95384:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2719.811874] LustreError: 93558:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2719.847694] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2719.908567] Lustre: lustre-MDT0000: new disk, initializing [ 2720.006984] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2720.024906] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2720.060639] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2721.127144] Lustre: 95408:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000d47cc9af x1875742720834176/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1788847917 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2723.658315] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2726.281182] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2726.296909] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2726.370318] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2748.391112] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2748.404865] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2748.421275] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2748.428776] Lustre: Skipped 1 previous similar message [ 2750.947475] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2750.948990] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2756.067478] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2756.079178] Lustre: Skipped 2 previous similar messages [ 2762.208130] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2762.406762] Lustre: server umount lustre-OST0000 complete [ 2766.317327] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2766.332780] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2766.337388] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2766.337397] Lustre: Skipped 2 previous similar messages [ 2766.348662] Lustre: Skipped 1 previous similar message [ 2772.365772] Lustre: server umount lustre-MDT0000 complete [ 2776.548794] LustreError: 93542:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2776.581642] LustreError: 93542:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2791.392273] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2792.053573] Lustre: server umount lustre-MDT0001 complete [ 2796.135105] Lustre: server umount MGS complete [ 2806.685954] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 02:13:05 (1788847985) [ 2826.395680] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 2830.834503] Key type lgssc unregistered [ 2831.145719] LNet: 97907:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2831.157424] LNetError: 97907:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2831.185624] LNet: Removed LNI 192.168.203.104@tcp [ 2832.928628] Key type .llcrypt unregistered [ 2832.932619] Key type ._llcrypt unregistered [ 2853.219270] Key type ._llcrypt registered [ 2853.221527] Key type .llcrypt registered [ 2853.383726] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 2854.538634] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2854.558580] alg: No test for adler32 (adler32-zlib) [ 2855.702282] Lustre: Lustre: Build Version: 2.17.57_108_g4a8ba26 [ 2856.130810] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 2857.872260] Key type lgssc registered [ 2859.025253] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2866.936881] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2866.947992] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2868.477126] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2873.868651] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2878.960775] LustreError: 99135:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2878.988715] LustreError: 99135:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2883.116674] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2883.556972] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2888.722934] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2896.644840] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2902.206173] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2909.089785] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2909.434197] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2913.538975] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 2916.514570] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2924.294573] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2939.370495] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2939.376342] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2939.391715] LustreError: Skipped 1 previous similar message [ 2939.392443] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2939.419829] Lustre: Skipped 1 previous similar message [ 2942.100970] Lustre: server umount lustre-OST0000 complete [ 2949.603647] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2949.616778] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2949.627635] Lustre: Skipped 1 previous similar message [ 2952.177367] Lustre: server umount lustre-MDT0000 complete [ 2954.722251] LustreError: 99135:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2954.765349] LustreError: 99135:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2956.396209] LustreError: 100980:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788848136 with bad export cookie 12305657434281030715 [ 2956.402018] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2956.420190] LustreError: 100980:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2956.849837] Lustre: server umount lustre-MDT0001 complete [ 2969.145521] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 2973.323473] Key type lgssc unregistered [ 2973.694797] LNet: 102667:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2973.711324] LNetError: 102667:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2973.729489] LNet: Removed LNI 192.168.203.104@tcp [ 2974.665106] Key type .llcrypt unregistered [ 2974.666676] Key type ._llcrypt unregistered [ 2988.864552] Key type ._llcrypt registered [ 2988.867741] Key type .llcrypt registered [ 2989.007929] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 2990.266500] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2990.315547] alg: No test for adler32 (adler32-zlib) [ 2991.489264] Lustre: Lustre: Build Version: 2.17.57_108_g4a8ba26 [ 2991.695306] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 2993.339474] Key type lgssc registered [ 2994.774221] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3001.965967] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3001.966300] LustreError: 103746:0:(tgt_mount.c:2570:server_fill_super()) cfs_race id 716 sleeping [ 3001.984987] Lustre: Skipped 1 previous similar message [ 3001.985043] LustreError: 103743:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 waking [ 3002.017379] LustreError: 103746:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 awake: rc=5000 [ 3002.024950] LustreError: 103746:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3002.040275] LustreError: 103746:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3002.042334] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3003.659329] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3012.461486] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3012.862064] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3017.843890] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 3022.734266] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing lsmod [ 3025.650714] LustreError: 104950:0:(tgt_mount.c:2570:server_fill_super()) cfs_race id 716 sleeping [ 3025.665675] LustreError: 104951:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 waking [ 3025.669611] LustreError: 104950:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 awake: rc=4989 [ 3025.674383] LustreError: 104950:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3025.682586] LustreError: 104950:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3025.685906] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3025.967948] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3031.010930] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3031.014473] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3031.039464] Lustre: Skipped 1 previous similar message [ 3031.060557] Lustre: Skipped 2 previous similar messages [ 3036.130087] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3036.138776] Lustre: Skipped 2 previous similar messages [ 3038.202049] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3038.217305] Lustre: Skipped 2 previous similar messages [ 3043.296507] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3043.299579] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3043.308568] Lustre: Skipped 2 previous similar messages [ 3043.528361] Lustre: server umount lustre-MDT0000 complete [ 3047.880872] LustreError: 103761:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788848227 with bad export cookie 15178613978940027876 [ 3047.883424] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3047.901957] LustreError: 103761:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3048.426248] LustreError: 103778:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3048.432621] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3048.443124] LustreError: 103778:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3048.485652] Lustre: Skipped 1 previous similar message [ 3048.493994] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3048.504660] LustreError: 104213:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000e2ac5aa6 x1875743259775872/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3048.695724] Lustre: server umount lustre-MDT0001 complete [ 3054.923748] Lustre: server umount lustre-OST0000 complete [ 3064.714616] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3065.359147] LustreError: 106017:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3065.512837] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3070.596347] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 3070.951728] LustreError: 106018:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3075.062904] LustreError: 106017:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3079.798098] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3080.205606] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3085.435038] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 3093.945409] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3100.590772] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3108.611552] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3108.983808] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3111.026459] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3114.975206] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 3122.795644] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3130.858872] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3130.860095] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3130.877542] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3131.874840] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3145.184411] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3145.340693] Lustre: server umount lustre-OST0000 complete [ 3151.334573] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3151.341239] Lustre: Skipped 1 previous similar message [ 3151.344912] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3151.352542] Lustre: Skipped 6 previous similar messages [ 3154.925028] Lustre: server umount lustre-MDT0000 complete [ 3156.453327] LustreError: 106017:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3156.476456] LustreError: 106017:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3159.037346] LustreError: 105999:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788848339 with bad export cookie 15178613978940028555 [ 3159.038634] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3159.053340] LustreError: 105999:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3159.513179] Lustre: server umount lustre-MDT0001 complete [ 3172.032355] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 3175.220320] Key type lgssc unregistered [ 3175.543426] LNet: 109550:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3175.549611] LNetError: 109550:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3175.574804] LNet: Removed LNI 192.168.203.104@tcp [ 3176.399193] Key type .llcrypt unregistered [ 3176.404867] Key type ._llcrypt unregistered [ 3200.956304] Key type ._llcrypt registered [ 3200.959169] Key type .llcrypt registered [ 3201.047242] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 3201.861153] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3201.921712] alg: No test for adler32 (adler32-zlib) [ 3203.175717] Lustre: Lustre: Build Version: 2.17.57_108_g4a8ba26 [ 3203.455791] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 3205.144229] Key type lgssc registered [ 3206.374034] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3214.768850] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3214.791787] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3216.478075] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3221.493387] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 3226.596977] LustreError: 110979:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3226.611954] LustreError: 110979:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3231.017842] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3231.371898] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3236.529727] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 3244.519718] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3250.660263] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3257.949781] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3258.383522] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3261.519831] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 3265.809373] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 3274.040723] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3282.430286] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3286.060590] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3292.133601] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3292.141428] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3292.172034] Lustre: Skipped 1 previous similar message [ 3292.187445] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3292.189531] Lustre: Skipped 1 previous similar message [ 3297.114665] Lustre: server umount lustre-OST0000 complete [ 3302.379247] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3302.387909] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3302.391607] Lustre: Skipped 1 previous similar message [ 3307.492464] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3307.501343] Lustre: Skipped 2 previous similar messages [ 3307.855702] Lustre: server umount lustre-MDT0000 complete [ 3312.189508] LustreError: 110961:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788848492 with bad export cookie 6855430989673064336 [ 3312.198818] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3312.201434] LustreError: 110961:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3312.896510] Lustre: server umount lustre-MDT0001 complete [ 3325.661955] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 3329.840268] Key type lgssc unregistered [ 3330.123293] LNet: 114615:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3330.132228] LNetError: 114615:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3330.152190] LNet: Removed LNI 192.168.203.104@tcp [ 3331.051904] Key type .llcrypt unregistered [ 3331.054449] Key type ._llcrypt unregistered [ 3352.981713] Key type ._llcrypt registered [ 3352.983887] Key type .llcrypt registered [ 3353.197479] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 3354.506530] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3354.618137] alg: No test for adler32 (adler32-zlib) [ 3355.674326] Lustre: Lustre: Build Version: 2.17.57_108_g4a8ba26 [ 3355.906126] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 3357.632233] Key type lgssc registered [ 3359.242458] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3367.785639] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3367.808602] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3369.249774] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3373.647252] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 3379.683220] LustreError: 115835:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3379.728572] LustreError: 115835:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3384.807130] LustreError: 115834:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3385.094437] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3385.649299] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3392.016949] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 3402.655946] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3408.857558] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3416.333872] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3416.764654] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3423.018740] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 3431.510799] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3439.334433] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3454.947860] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3454.953667] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3454.962833] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3454.971728] Lustre: Skipped 1 previous similar message [ 3454.981491] Lustre: Skipped 1 previous similar message [ 3457.296844] Lustre: server umount lustre-OST0000 complete [ 3462.626359] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3462.635467] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3462.650315] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3465.185452] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3465.192177] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3467.787503] Lustre: server umount lustre-MDT0000 complete [ 3470.307519] LustreError: 115830:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3470.331333] LustreError: 115830:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3471.862408] LustreError: 115814:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788848651 with bad export cookie 13623584797993642472 [ 3471.885469] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3472.142754] Lustre: server umount lustre-MDT0001 complete [ 3484.463626] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 3487.886431] Key type lgssc unregistered [ 3488.245795] LNet: 119388:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3488.263807] LNetError: 119388:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3488.281956] LNet: Removed LNI 192.168.203.104@tcp [ 3489.354645] Key type .llcrypt unregistered [ 3489.362852] Key type ._llcrypt unregistered [ 3514.663837] Key type ._llcrypt registered [ 3514.672030] Key type .llcrypt registered [ 3514.798691] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 3515.719679] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3515.743798] alg: No test for adler32 (adler32-zlib) [ 3516.803954] Lustre: Lustre: Build Version: 2.17.57_108_g4a8ba26 [ 3517.066077] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 3518.768471] Key type lgssc registered [ 3520.186817] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3527.639309] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3527.685084] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3529.253220] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3535.191130] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 3539.429235] LustreError: 120799:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3539.440107] LustreError: 120799:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3544.560555] LustreError: 120800:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3545.184866] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3545.500642] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3550.982337] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 3560.650689] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3566.613057] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3574.407599] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3574.820579] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3582.657112] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 3592.173724] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3599.434118] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3608.332603] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3612.413731] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3612.420345] Lustre: Skipped 2 previous similar messages [ 3616.944501] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3616.949879] Lustre: Skipped 2 previous similar messages [ 3618.399955] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3618.452922] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3632.089544] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.203.4@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3632.102238] Lustre: Skipped 2 previous similar messages [ 3641.486815] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.203.4@tcp 192.168.0.[2,10]@tcp [ 3646.921671] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.203.4/24@tcp in log lustre-MDT0000 [ 3646.929601] Lustre: Skipped 2 previous similar messages [ 3657.345831] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.4/24@tcp [ 3657.353267] Lustre: Skipped 1 previous similar message [ 3665.889754] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3665.897517] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3665.910732] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3665.921256] Lustre: Skipped 1 previous similar message [ 3665.934225] Lustre: Skipped 1 previous similar message [ 3671.015851] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3671.021966] Lustre: Skipped 1 previous similar message [ 3676.133560] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3676.145848] Lustre: Skipped 1 previous similar message [ 3678.176168] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3678.352988] Lustre: server umount lustre-OST0000 complete [ 3683.811166] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3683.815436] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3683.840428] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3686.371746] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3688.860469] Lustre: server umount lustre-MDT0000 complete [ 3691.490087] LustreError: 120800:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3691.506023] LustreError: 120800:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3693.262746] LustreError: 124555:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788848873 with bad export cookie 5092662221370317453 [ 3693.270341] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3693.279656] LustreError: 124555:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3693.856943] Lustre: server umount lustre-MDT0001 complete [ 3707.272805] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 3711.033653] Key type lgssc unregistered [ 3711.397056] LNet: 125906:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3711.431572] LNetError: 125906:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3712.492564] LNet: Removed LNI 192.168.203.104@tcp [ 3713.726853] Key type .llcrypt unregistered [ 3713.729839] Key type ._llcrypt unregistered [ 3743.937425] Key type ._llcrypt registered [ 3743.944136] Key type .llcrypt registered [ 3744.087732] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 3745.194252] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3745.320068] alg: No test for adler32 (adler32-zlib) [ 3746.409670] Lustre: Lustre: Build Version: 2.17.57_108_g4a8ba26 [ 3746.680863] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 3748.384409] Key type lgssc registered [ 3749.591956] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3757.232080] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3757.272508] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3758.786476] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3758.818251] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3758.828202] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.203.4/24@tcp [ 3763.771761] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 3768.300910] LustreError: 127469:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3768.337905] LustreError: 127469:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3773.266616] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3773.419874] LustreError: 127470:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3773.669602] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3773.721983] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.4/24@tcp [ 3773.762614] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3779.284515] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 3787.490684] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3793.850520] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3800.416780] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3800.825612] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3803.917541] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 3808.113990] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 3816.565285] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3827.180755] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3827.187622] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3827.196854] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3828.706694] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3828.717611] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3833.227955] Lustre: server umount lustre-OST0000 complete [ 3838.952218] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3838.968099] Lustre: Skipped 1 previous similar message [ 3838.971230] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3838.981642] Lustre: Skipped 1 previous similar message [ 3844.066657] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3844.075782] Lustre: Skipped 1 previous similar message [ 3848.168167] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3848.172678] Lustre: Skipped 1 previous similar message [ 3851.232148] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3851.404307] Lustre: server umount lustre-MDT0000 complete [ 3853.282365] LustreError: 127463:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3853.306954] LustreError: 127463:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3855.622884] LustreError: 127450:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788849035 with bad export cookie 9185268826458302085 [ 3855.629453] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3855.640979] LustreError: 127450:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3856.091723] Lustre: server umount lustre-MDT0001 complete [ 3868.083884] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 3871.073701] Key type lgssc unregistered [ 3871.437342] LNet: 131059:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3871.446127] LNetError: 131059:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3871.470164] LNet: Removed LNI 192.168.203.104@tcp [ 3872.458359] Key type .llcrypt unregistered [ 3872.460102] Key type ._llcrypt unregistered [ 3901.759732] Key type ._llcrypt registered [ 3901.762251] Key type .llcrypt registered [ 3901.871518] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_hostid [ 3918.440277] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 3919.577145] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3919.726802] alg: No test for adler32 (adler32-zlib) [ 3920.893391] Lustre: Lustre: Build Version: 2.17.57_108_g4a8ba26 [ 3921.239532] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 3922.984175] Key type lgssc registered [ 3924.335422] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3961.331520] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3961.369788] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3962.652552] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3962.708831] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3962.826389] Lustre: lustre-MDT0000: new disk, initializing [ 3962.938943] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3962.964539] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3967.768892] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 3978.894988] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3979.069767] Lustre: 134456:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3979.144792] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3979.152732] Lustre: Skipped 1 previous similar message [ 3979.325800] Lustre: lustre-MDT0001: new disk, initializing [ 3979.457472] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3979.501766] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3979.518833] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3984.335318] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 3994.827770] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4001.002356] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4008.491546] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4008.887486] Lustre: lustre-OST0000: new disk, initializing [ 4008.899701] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4008.912296] Lustre: 136425:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4009.052490] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4016.227336] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4016.257647] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4016.280809] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 4016.400912] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4025.850428] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4030.941124] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4031.349560] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4036.176783] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4036.424579] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4042.216079] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4042.217533] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4042.224538] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4042.236732] Lustre: Skipped 1 previous similar message [ 4044.912834] Lustre: server umount lustre-OST0000 complete [ 4051.427297] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4051.444805] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4051.497619] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4051.505522] Lustre: Skipped 1 previous similar message [ 4052.452249] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4055.453313] Lustre: server umount lustre-MDT0000 complete [ 4058.827455] LustreError: 136423:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788849238 with bad export cookie 7549525342329292538 [ 4058.828111] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4058.836082] LustreError: 136423:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4059.399865] Lustre: server umount lustre-MDT0001 complete [ 4067.540430] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4067.990033] LustreError: 138641:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4068.009346] LustreError: 138641:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4068.093375] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4072.776873] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 4073.444970] LustreError: 138642:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4073.465882] LustreError: 138642:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4078.563812] LustreError: 138641:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4081.448223] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4081.905115] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4087.462602] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 4095.883619] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4102.241364] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4109.434824] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4109.772203] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4116.284531] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 4123.805774] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4128.778465] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4129.112616] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4133.923794] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4134.262630] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4142.611106] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4142.983860] Lustre: lustre-OST0001: new disk, initializing [ 4142.992285] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4142.999682] Lustre: 142015:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4143.091838] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4149.410429] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 4152.960368] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4152.970656] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4153.117697] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4157.292890] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4157.476339] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4161.877330] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4162.098465] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4167.759628] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9c2346635000.ost_server_uuid 50 [ 4169.132910] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9c2346635000.ost_server_uuid in IDLE state after 0 sec [ 4178.922957] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4178.925304] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4178.930429] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4178.930438] Lustre: Skipped 2 previous similar messages [ 4178.942797] LustreError: Skipped 1 previous similar message [ 4178.967675] Lustre: Skipped 1 previous similar message [ 4184.055687] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4184.065729] Lustre: Skipped 2 previous similar messages [ 4189.158793] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4192.224138] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4192.432945] Lustre: server umount lustre-OST0001 complete [ 4194.277462] LustreError: 140468:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4194.301138] LustreError: 140468:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4197.859640] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4197.873491] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4197.889674] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4197.896646] Lustre: Skipped 1 previous similar message [ 4199.407772] LustreError: 141487:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4199.429452] LustreError: 141487:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4209.643478] LustreError: 140466:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4209.662176] LustreError: 140466:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4210.147274] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4210.359448] Lustre: server umount lustre-OST0000 complete [ 4214.755745] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4214.771058] Lustre: Skipped 2 previous similar messages [ 4214.773092] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4214.782848] Lustre: Skipped 7 previous similar messages [ 4220.235281] Lustre: server umount lustre-MDT0000 complete [ 4223.334908] LustreError: 142014:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788849403 with bad export cookie 7549525342329293427 [ 4223.340596] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4223.347576] LustreError: 142014:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4223.662669] Lustre: server umount lustre-MDT0001 complete [ 4240.614028] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 4243.075804] Key type lgssc unregistered [ 4243.315434] LNet: 145018:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4243.334189] LNetError: 145018:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4243.355592] LNet: Removed LNI 192.168.203.104@tcp [ 4244.180552] Key type .llcrypt unregistered [ 4244.182464] Key type ._llcrypt unregistered [ 4278.867195] Key type ._llcrypt registered [ 4278.871615] Key type .llcrypt registered [ 4279.011972] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_hostid [ 4293.579538] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 4294.973683] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4295.013723] alg: No test for adler32 (adler32-zlib) [ 4296.015950] Lustre: Lustre: Build Version: 2.17.57_108_g4a8ba26 [ 4296.291805] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 4298.011647] Key type lgssc registered [ 4299.044031] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4331.189333] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4331.212848] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4332.452597] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4332.476315] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4332.559846] Lustre: lustre-MDT0000: new disk, initializing [ 4332.616287] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4332.629667] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4336.961536] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 4347.420921] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4347.551887] Lustre: 149176:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4347.606238] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4347.610111] Lustre: Skipped 1 previous similar message [ 4347.671494] Lustre: lustre-MDT0001: new disk, initializing [ 4347.778950] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4347.840758] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4347.866311] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4352.134774] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 4360.707223] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4365.680337] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4371.545547] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4371.773907] Lustre: lustre-OST0000: new disk, initializing [ 4371.777245] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4371.783194] Lustre: 151143:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4371.865833] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4377.075733] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 4378.208646] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4378.219154] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4378.330322] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4385.303886] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4389.921231] Lustre: Failing over lustre-OST0000 [ 4392.115887] Lustre: server umount lustre-OST0000 complete [ 4408.273729] Lustre: 147392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788849572/real 1788849572] req@00000000ec54011b x1875744628178176/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1788849588 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4408.300772] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4409.312182] Lustre: 147395:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788849573/real 1788849573] req@0000000049242900 x1875744628178432/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788849589 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4409.355083] Lustre: 147395:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4409.366344] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4409.754106] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4410.135300] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4410.158531] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4415.201821] Lustre: 147395:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788849579/real 1788849579] req@0000000085a4a858 x1875744628179968/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788849595 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4415.504250] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 4418.276941] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4418.348909] Lustre: Failing over lustre-MDT0000 [ 4418.357500] LustreError: 147391:0:(import.c:707:ptlrpc_connect_import_locked()) can't connect to a closed import [ 4418.554467] Lustre: server umount lustre-MDT0000 complete [ 4419.429955] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4419.460463] Lustre: Skipped 2 previous similar messages [ 4419.859861] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 4419.860780] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4420.516405] Lustre: 147394:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788849584/real 1788849584] req@00000000a09fb164 x1875744628181120/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788849600 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4420.548613] Lustre: 147394:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4424.672148] Lustre: 147394:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788849588/real 1788849588] req@000000000dfe8be9 x1875744628182144/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788849604 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4424.677844] LustreError: 149190:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4424.700660] LustreError: 149190:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 4429.795350] LustreError: 149184:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4429.820376] LustreError: 149184:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4434.914509] LustreError: 149186:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4434.942207] LustreError: 149186:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4436.129342] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4436.308851] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4436.487021] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 4436.536779] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4436.587412] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4440.292049] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4440.641918] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 4441.578162] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4441.604106] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 4441.653324] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4450.785674] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4450.797530] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4450.815206] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4451.812925] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4451.828091] Lustre: Skipped 1 previous similar message [ 4456.933691] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4456.943714] Lustre: Skipped 1 previous similar message [ 4462.049463] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4462.058526] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4462.063405] Lustre: Skipped 1 previous similar message [ 4462.176477] Lustre: server umount lustre-OST0000 complete [ 4467.172261] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4467.174848] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4467.195892] Lustre: Skipped 1 previous similar message [ 4467.198078] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4467.213761] Lustre: Skipped 1 previous similar message [ 4471.243478] Lustre: server umount lustre-MDT0000 complete [ 4472.290516] LustreError: 149189:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4472.305394] LustreError: 149189:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4474.554666] LustreError: 149168:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788849654 with bad export cookie 3759226468399643160 [ 4474.558272] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4474.571623] LustreError: 149168:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4474.923627] Lustre: server umount lustre-MDT0001 complete [ 4483.859528] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 4486.072189] Key type lgssc unregistered [ 4486.410152] LNet: 154799:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4486.420693] LNetError: 154799:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4486.443657] LNet: Removed LNI 192.168.203.104@tcp [ 4487.241507] Key type .llcrypt unregistered [ 4487.243871] Key type ._llcrypt unregistered [ 4519.257416] Key type ._llcrypt registered [ 4519.260056] Key type .llcrypt registered [ 4519.357535] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 4520.566105] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4520.588550] alg: No test for adler32 (adler32-zlib) [ 4521.591776] Lustre: Lustre: Build Version: 2.17.57_108_g4a8ba26 [ 4521.845271] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 4523.512348] Key type lgssc registered [ 4524.382147] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4530.702359] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4530.732091] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4532.168253] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4535.771939] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 4542.434033] LustreError: 156315:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4542.445909] LustreError: 156315:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4543.553604] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4546.974274] LustreError: 156343:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4546.986292] LustreError: 156343:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4546.992813] LustreError: 156343:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 14, retries 0, failed: rc = -5 [ 4547.372372] Lustre: server umount lustre-MDT0000 complete [ 4559.530794] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_hostid [ 4568.396380] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 4594.692638] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4594.863678] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4594.893591] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4594.946316] Lustre: lustre-MDT0000: new disk, initializing [ 4594.996231] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4595.011662] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4598.309914] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 4606.244677] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4606.315835] Lustre: 159645:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4606.357475] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4606.362299] Lustre: Skipped 1 previous similar message [ 4606.425289] Lustre: lustre-MDT0001: new disk, initializing [ 4606.483157] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4606.516287] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4606.522858] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4609.611792] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 4615.963853] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4620.106645] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4624.764538] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4625.015688] Lustre: lustre-OST0000: new disk, initializing [ 4625.021657] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4625.025671] Lustre: 161610:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4625.088994] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4629.466188] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 4632.622460] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4632.641565] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4632.701711] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4636.208620] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4640.154744] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4641.529363] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 02:43:40 (1788849820) [ 4831.715291] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4831.726093] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4831.738615] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4834.928866] Lustre: server umount lustre-MDT0000 complete [ 4837.208264] LustreError: 161609:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788850017 with bad export cookie 8551763124389847406 [ 4837.208747] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4837.214096] LustreError: 161609:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4837.344693] LustreError: 163376:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4837.347414] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4837.374688] Lustre: Skipped 2 previous similar messages [ 4837.379901] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4837.382523] Lustre: Skipped 2 previous similar messages [ 4842.465856] LustreError: 162130:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4842.466039] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4843.484693] Lustre: server umount lustre-MDT0001 complete [ 4860.384128] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4860.459747] Lustre: server umount lustre-OST0000 complete [ 4865.964168] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_hostid [ 4872.374148] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 4895.603810] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4895.793406] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4895.829191] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4895.891500] Lustre: lustre-MDT0000: new disk, initializing [ 4895.942852] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4895.954687] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4899.139060] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 4906.423681] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4906.483380] Lustre: 166257:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4906.528161] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4906.531366] Lustre: Skipped 1 previous similar message [ 4906.600916] Lustre: lustre-MDT0001: new disk, initializing [ 4906.671802] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4906.697096] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4906.706580] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4909.428378] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 4915.876306] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4919.469455] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4923.738891] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4923.928418] Lustre: lustre-OST0000: new disk, initializing [ 4923.933192] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4923.938048] Lustre: 168223:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4923.995311] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4927.503905] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 4930.582312] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4930.589891] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4930.641227] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4932.967821] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4935.995192] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4936.148888] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4938.906272] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4939.066535] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4940.773801] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4940.777146] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4940.781214] LustreError: Skipped 1 previous similar message [ 4940.782300] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4940.793815] Lustre: Skipped 1 previous similar message [ 4945.891521] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4945.897822] Lustre: Skipped 2 previous similar messages [ 4946.380590] Lustre: server umount lustre-OST0000 complete [ 4951.014750] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4954.841587] Lustre: server umount lustre-MDT0000 complete [ 4956.131188] LustreError: 167058:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4956.139104] LustreError: 167058:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4957.023698] LustreError: 166250:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788850137 with bad export cookie 8551763124390006236 [ 4957.025814] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4957.031619] LustreError: 166250:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4957.245371] Lustre: server umount lustre-MDT0001 complete [ 4962.110696] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 02:49:01 (1788850141) [ 4971.670802] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_hostid [ 4977.641091] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 5000.266051] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5000.388158] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5000.405157] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5000.448173] Lustre: lustre-MDT0000: new disk, initializing [ 5000.503466] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5000.516121] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5002.970304] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 5008.948886] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5009.027108] Lustre: 172975:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5009.202269] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5011.749328] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 5017.102702] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5020.315928] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5024.048256] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5024.222435] Lustre: 174949:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5027.360956] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 5031.970862] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5032.052127] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5032.574124] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5037.030371] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5037.031914] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5037.032399] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5037.032404] Lustre: Skipped 2 previous similar messages [ 5037.037838] Lustre: Skipped 2 previous similar messages [ 5041.851499] Lustre: server umount lustre-OST0000 complete [ 5045.218661] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5045.223153] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5050.072121] Lustre: server umount lustre-MDT0000 complete [ 5052.310959] LustreError: 172967:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788850232 with bad export cookie 8551763124390007118 [ 5052.312257] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5052.317054] LustreError: 172967:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5052.496994] Lustre: server umount lustre-MDT0001 complete [ 5057.096280] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 02:50:36 (1788850236) [ 5065.233639] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_hostid [ 5070.002888] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 5087.977868] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5088.117876] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5088.134372] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5088.137382] Lustre: Skipped 3 previous similar messages [ 5088.178840] Lustre: lustre-MDT0000: new disk, initializing [ 5088.181228] Lustre: Skipped 2 previous similar messages [ 5088.222545] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5088.227728] Lustre: Skipped 2 previous similar messages [ 5088.237212] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5088.241280] Lustre: Skipped 2 previous similar messages [ 5090.511895] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 5096.362317] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5096.438272] Lustre: 179262:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5096.637404] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5098.676157] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 5103.932341] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5106.528722] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5109.683919] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5109.811336] Lustre: 181236:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5112.641321] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 5116.999685] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5117.433738] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5122.017305] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5122.021403] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5122.024944] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5122.027706] Lustre: Skipped 2 previous similar messages [ 5122.034204] Lustre: Skipped 4 previous similar messages [ 5126.828396] Lustre: server umount lustre-OST0000 complete [ 5134.656537] Lustre: server umount lustre-MDT0000 complete [ 5136.484806] LustreError: 179252:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788850316 with bad export cookie 8551763124390008189 [ 5136.487113] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5136.500676] LustreError: 179252:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5136.653191] Lustre: server umount lustre-MDT0001 complete [ 5142.007331] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 5143.507784] Key type lgssc unregistered [ 5143.685234] LNet: 183087:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5143.690413] LNetError: 183087:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5143.705106] LNet: Removed LNI 192.168.203.104@tcp [ 5144.225756] Key type .llcrypt unregistered [ 5144.227682] Key type ._llcrypt unregistered [ 5157.299259] Key type ._llcrypt registered [ 5157.300902] Key type .llcrypt registered [ 5157.346234] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 5157.768223] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5157.832520] alg: No test for adler32 (adler32-zlib) [ 5158.705325] Lustre: Lustre: Build Version: 2.17.57_108_g4a8ba26 [ 5158.817402] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 5160.416159] Key type lgssc registered [ 5160.909525] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5164.795659] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5164.812953] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5166.049680] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5168.193285] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 5172.400411] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5172.591265] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5174.517298] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 5178.289758] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5180.910821] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5184.331867] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5184.500469] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5187.407933] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 5191.022110] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5191.662410] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5217.250399] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5217.257330] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5217.266198] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5218.276089] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5218.276752] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5218.284200] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5219.573302] Lustre: server umount lustre-OST0000 complete [ 5223.395173] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5223.399236] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5223.408299] Lustre: Skipped 1 previous similar message [ 5223.415164] Lustre: Skipped 2 previous similar messages [ 5227.635155] Lustre: server umount lustre-MDT0000 complete [ 5228.515817] LustreError: 186121:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5228.526491] LustreError: 186121:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 5229.723405] LustreError: 184458:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788850409 with bad export cookie 8614703709011654323 [ 5229.726050] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5229.730168] LustreError: 184458:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5229.891813] Lustre: server umount lustre-MDT0001 complete [ 5235.449466] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 5236.884446] Key type lgssc unregistered [ 5237.058195] LNet: 188028:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5237.064979] LNetError: 188028:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5237.079428] LNet: Removed LNI 192.168.203.104@tcp [ 5237.447311] Key type .llcrypt unregistered [ 5237.449238] Key type ._llcrypt unregistered [ 5250.050678] Key type ._llcrypt registered [ 5250.052482] Key type .llcrypt registered [ 5250.099617] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 5250.549471] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5250.572844] alg: No test for adler32 (adler32-zlib) [ 5251.455681] Lustre: Lustre: Build Version: 2.17.57_108_g4a8ba26 [ 5251.587286] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 5253.208206] Key type lgssc registered [ 5253.732311] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5257.354921] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5257.367140] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5258.570677] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5260.337526] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 5264.045213] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5264.236802] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5265.941896] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 5269.100649] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5271.538175] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5274.372985] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5274.465735] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 5274.515523] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5276.980105] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 5279.975381] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5283.816060] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5288.934049] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5288.937557] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5288.943406] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5288.948892] Lustre: Skipped 1 previous similar message [ 5290.744402] Lustre: lustre-OST0000: Not available for connect from 192.168.203.4@tcp (stopping) [ 5294.051089] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5294.055717] Lustre: Skipped 1 previous similar message [ 5294.193381] Lustre: server umount lustre-OST0000 complete [ 5297.675947] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff9c23466cf800.ost_server_uuid 50 [ 5298.404861] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5309.409818] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5309.416145] Lustre: Skipped 1 previous similar message [ 5309.419650] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5309.422276] Lustre: Skipped 1 previous similar message [ 5314.915424] Lustre: server umount lustre-MDT0000 complete [ 5316.664280] LustreError: 189398:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788850496 with bad export cookie 7754063872919515986 [ 5316.665083] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5316.673277] LustreError: 189398:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5316.826283] Lustre: server umount lustre-MDT0001 complete [ 5320.254056] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 02:54:59 (1788850499) [ 5322.850452] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5323.059616] LustreError: 193308:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5323.068556] LustreError: 193308:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5323.093756] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5324.810424] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 5328.354218] LustreError: 193309:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5328.584460] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5328.763757] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5330.481071] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 5333.429873] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5335.698693] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5338.469749] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5338.637108] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5341.110602] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 5344.176382] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5347.160185] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5347.249950] Lustre: lustre-OST0001: new disk, initializing [ 5347.252878] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5347.257161] Lustre: 196146:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5348.330293] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5348.427839] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5348.433889] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5348.481150] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5349.758776] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 5354.517867] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5358.561085] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5358.565653] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5358.571317] Lustre: Skipped 1 previous similar message [ 5358.575509] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5358.581841] Lustre: Skipped 2 previous similar messages [ 5361.736821] Lustre: server umount lustre-OST0000 complete [ 5364.198909] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5364.199916] LustreError: 196285:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5364.215888] LustreError: 196285:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5366.182878] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5366.282990] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5369.315945] LustreError: 195127:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5369.324801] LustreError: 195127:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5374.438317] LustreError: 195128:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5374.450057] LustreError: 195128:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5379.556263] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5379.556637] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5379.559795] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5379.564537] Lustre: Skipped 2 previous similar messages [ 5379.569344] Lustre: Skipped 4 previous similar messages [ 5382.743566] Lustre: server umount lustre-OST0001 complete [ 5384.674492] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5390.450470] Lustre: server umount lustre-MDT0000 complete [ 5392.016537] LustreError: 193288:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788850572 with bad export cookie 7754063872919517057 [ 5392.020087] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5392.022221] LustreError: 193288:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5392.170208] Lustre: server umount lustre-MDT0001 complete [ 5402.221444] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 02:56:21 (1788850581) [ 5404.834855] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5404.968733] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5404.988969] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5405.078473] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5405.081744] Lustre: Skipped 1 previous similar message [ 5406.760436] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 5410.365159] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5410.425733] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5410.451869] Lustre: 199542:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5412.251208] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 5415.203319] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5417.500204] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5420.174281] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5420.256138] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5422.643232] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 5424.346851] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 5425.634693] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5428.592249] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5428.628300] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5431.219898] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 5432.489545] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 5434.353952] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5437.928756] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5437.932242] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5437.934932] Lustre: Skipped 2 previous similar messages [ 5437.939884] Lustre: Skipped 5 previous similar messages [ 5441.604916] Lustre: server umount lustre-OST0000 complete [ 5443.045439] LustreError: 201391:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5443.053570] LustreError: 201391:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 5453.282272] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5453.285745] LustreError: Skipped 2 previous similar messages [ 5458.779359] Lustre: server umount lustre-OST0001 complete [ 5461.985032] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5461.988875] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5461.994776] Lustre: Skipped 2 previous similar messages [ 5466.335547] Lustre: server umount lustre-MDT0000 complete [ 5467.659610] LustreError: 199534:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788850647 with bad export cookie 7754063872919518415 [ 5467.662619] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5467.664378] LustreError: 199534:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5467.789916] Lustre: server umount lustre-MDT0001 complete [ 5477.429740] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 02:57:36 (1788850656) [ 5490.445288] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_hostid [ 5498.212173] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 5536.390486] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5536.890426] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5536.972127] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5537.131986] Lustre: lustre-MDT0000: new disk, initializing [ 5537.213464] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5537.233357] Lustre: Skipped 3 previous similar messages [ 5537.255501] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5544.139179] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 5557.439380] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5557.577552] Lustre: 207617:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5557.633177] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5557.643870] Lustre: Skipped 1 previous similar message [ 5557.772859] Lustre: lustre-MDT0001: new disk, initializing [ 5558.021569] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5558.043114] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5563.673596] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 5571.669204] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5576.142905] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5581.161687] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5581.357755] Lustre: lustre-OST0000: new disk, initializing [ 5581.360937] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5581.365940] Lustre: 209581:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5585.600225] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 5590.562676] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5590.571190] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5590.657599] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5591.537796] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5594.558935] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5594.682067] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5597.788985] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5597.952782] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5600.737130] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5600.741756] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5600.745044] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5600.754860] Lustre: Skipped 2 previous similar messages [ 5600.760715] Lustre: Skipped 9 previous similar messages [ 5605.588876] Lustre: server umount lustre-OST0000 complete [ 5614.432263] Lustre: server umount lustre-MDT0000 complete [ 5616.101982] LustreError: 207621:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5616.113800] LustreError: 207621:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 5616.996124] LustreError: 207609:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788850796 with bad export cookie 7754063872919520186 [ 5617.002492] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5617.005875] LustreError: 207609:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5617.226713] Lustre: server umount lustre-MDT0001 complete [ 5622.783620] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5623.158965] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5623.166398] Lustre: Skipped 2 previous similar messages [ 5625.937463] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 5631.224153] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5634.186780] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 5639.530506] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5644.293309] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5653.978930] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5665.649054] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 5679.699433] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5687.144102] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5687.755368] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5695.511900] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5695.964232] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5699.554108] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5699.592628] Lustre: Skipped 2 previous similar messages [ 5712.352128] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5712.533575] Lustre: server umount lustre-OST0000 complete [ 5718.412902] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5722.760911] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5751.352325] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5751.691965] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5751.702909] Lustre: Skipped 2 previous similar messages [ 5758.465044] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 5764.972953] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5764.987945] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5764.997722] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5765.013958] Lustre: Skipped 1 previous similar message [ 5765.693775] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5770.387778] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 3 sec [ 5775.331464] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5775.352819] LustreError: Skipped 1 previous similar message [ 5775.358126] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5775.385209] Lustre: Skipped 10 previous similar messages [ 5780.259239] Lustre: server umount lustre-OST0000 complete [ 5790.418796] Lustre: server umount lustre-MDT0000 complete [ 5790.695788] LustreError: 211793:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5790.706292] LustreError: 211793:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 5794.211968] LustreError: 214918:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788850974 with bad export cookie 7754063872919521068 [ 5794.219839] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5794.230323] LustreError: 214918:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5794.504098] Lustre: server umount lustre-MDT0001 complete [ 5802.785727] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 03:03:01 (1788850981) [ 5809.658368] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5815.043740] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 5822.582768] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5826.989322] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 5834.268291] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5839.639526] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5846.359821] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5853.576219] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 5861.002036] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5866.109510] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5866.329564] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5871.434938] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5871.631763] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5878.737455] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5878.968866] Lustre: lustre-OST0001: new disk, initializing [ 5878.972066] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5878.980631] Lustre: 221144:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5885.899028] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 5889.172515] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5889.188633] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5889.249519] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5895.514717] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5899.859677] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5900.114437] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5904.440274] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5904.739668] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5909.992469] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5909.999177] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5909.999741] Lustre: Skipped 5 previous similar messages [ 5910.015954] LustreError: Skipped 1 previous similar message [ 5921.248339] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5921.339350] Lustre: server umount lustre-OST0001 complete [ 5925.389142] LustreError: 219600:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5925.432666] LustreError: 219600:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 5929.639385] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5929.939196] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5934.389798] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5934.624874] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5964.158854] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5969.905444] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 5969.921435] Lustre: lustre-OST0001-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5969.932722] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 5971.439880] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 5979.975914] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5981.887125] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5985.272454] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5985.283033] LustreError: Skipped 1 previous similar message [ 5990.165387] Lustre: server umount lustre-OST0001 complete [ 6022.112448] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6022.326100] Lustre: server umount lustre-OST0000 complete [ 6032.178635] Lustre: server umount lustre-MDT0000 complete [ 6036.465784] LustreError: 217757:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788851216 with bad export cookie 7754063872919522020 [ 6036.475208] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6036.480590] LustreError: 217757:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6043.187281] Lustre: server umount lustre-MDT0001 complete [ 6067.398535] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 03:07:25 (1788851245) [ 6073.304893] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6073.658606] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6073.706437] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6073.868405] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6073.878683] Lustre: Skipped 5 previous similar messages [ 6078.068367] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 6087.168918] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6087.236777] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6087.256462] Lustre: 226387:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6092.212434] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 6100.221356] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6105.525952] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6112.001829] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6112.278717] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6118.535321] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 6124.636489] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6133.713688] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6133.766055] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6140.154890] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 6147.944721] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6149.333815] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6154.654445] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6154.887115] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6160.919717] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6161.189296] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6167.474055] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9c2346d3e000.ost_server_uuid 50 [ 6169.357474] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9c2346d3e000.ost_server_uuid in IDLE state after 0 sec [ 6170.597069] Lustre: Permanently deactivating lustre-OST0001 [ 6170.620854] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6174.823920] Lustre: Permanently reactivating lustre-OST0001 [ 6174.846871] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 6174.853652] Lustre: Skipped 2 previous similar messages [ 6180.332347] LustreError: lustre-OST0001-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 6180.342792] LustreError: Skipped 1 previous similar message [ 6180.354368] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6180.373653] Lustre: Skipped 7 previous similar messages [ 6180.387731] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6180.399519] Lustre: Skipped 21 previous similar messages [ 6194.144575] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6194.264442] Lustre: server umount lustre-OST0001 complete [ 6199.280233] LustreError: 228232:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6199.300986] LustreError: 228232:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 37 previous similar messages [ 6203.977315] Lustre: server umount lustre-OST0000 complete [ 6222.307542] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6222.461964] Lustre: server umount lustre-MDT0000 complete [ 6226.258560] LustreError: 228248:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788851406 with bad export cookie 7754063872919523308 [ 6226.271368] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6226.273239] LustreError: 228248:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6226.609509] Lustre: server umount lustre-MDT0001 complete [ 6251.023561] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 03:10:29 (1788851429) [ 6257.366118] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6257.638499] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6257.664539] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6257.671391] Lustre: Skipped 2 previous similar messages [ 6261.314975] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 6268.299966] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6268.418660] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6268.455635] Lustre: 233088:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6273.131691] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 6280.531715] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6286.542294] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6293.583174] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6293.995092] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6294.026746] Lustre: Permanently deactivating lustre-OST0000 [ 6294.049788] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 6300.232900] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6300.357858] LustreError: 235262:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0000: cannot cleanup orphans: rc = -108 [ 6301.052315] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 6308.561869] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6315.450901] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6315.572660] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6321.842243] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 6323.986407] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 6328.327649] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6329.936266] Lustre: Permanently reactivating lustre-OST0000 [ 6329.947781] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 6329.951547] Lustre: Skipped 2 previous similar messages [ 6332.451434] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6332.453125] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 32 seconds. I think it's dead, and I am evicting it. exp 00000000e6fcf01c, cur 1788851512 deadline 1788851510 last 1788851480 [ 6332.458039] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6332.459716] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6332.459723] Lustre: Skipped 1 previous similar message [ 6332.520954] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6349.196045] Lustre: server umount lustre-OST0001 complete [ 6358.680659] Lustre: server umount lustre-OST0000 complete [ 6368.468463] Lustre: server umount lustre-MDT0000 complete [ 6372.418438] LustreError: 233839:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788851552 with bad export cookie 7754063872919524981 [ 6372.419192] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6372.428444] LustreError: 233839:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6372.967135] Lustre: server umount lustre-MDT0001 complete [ 6383.888672] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 6387.308413] Key type lgssc unregistered [ 6387.735491] LNet: 237962:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6387.748075] LNetError: 237962:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6387.763561] LNet: Removed LNI 192.168.203.104@tcp [ 6388.757834] Key type .llcrypt unregistered [ 6388.761114] Key type ._llcrypt unregistered [ 6416.056202] Key type ._llcrypt registered [ 6416.058130] Key type .llcrypt registered [ 6416.148720] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 6416.925834] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6416.994084] alg: No test for adler32 (adler32-zlib) [ 6418.006371] Lustre: Lustre: Build Version: 2.17.57_108_g4a8ba26 [ 6418.293739] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 6419.952211] Key type lgssc registered [ 6421.101609] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6432.031603] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6432.048784] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6433.613766] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6438.367885] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 6444.027815] LustreError: 239425:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6444.053573] LustreError: 239425:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6447.608775] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6447.820909] Lustre: Found index 1 for lustre-MDT0001, updating log [ 6447.840763] Lustre: 239414:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6447.858400] Lustre: Permanently deactivating lustre-MDT0001 [ 6447.869542] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 6448.146155] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6450.854962] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 6450.867645] LustreError: 239456:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 6450.891195] LustreError: 239456:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 6450.906543] LustreError: 239456:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 17, retries 1, failed: rc = -108 [ 6452.370186] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 6459.925737] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6466.295839] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6474.324366] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6474.589683] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6479.736317] LustreError: 241268:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6479.792811] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 6482.305820] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 6484.972302] LustreError: 241268:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6490.102466] LustreError: 241268:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6490.834426] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6493.238202] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 6497.528480] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6497.738522] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6502.954694] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 6502.997946] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 6503.722485] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 6510.398904] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6513.094483] Lustre: Permanently reactivating lustre-MDT0001 [ 6513.109215] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 6513.112847] Lustre: Skipped 2 previous similar messages [ 6518.043748] Lustre: lustre-MDT0001: Client 049d4d13-26c7-4736-bf6c-c34baaa38201 (at 192.168.203.4@tcp) reconnecting [ 6524.077694] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6525.468873] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6530.266266] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 6530.460136] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 6534.228278] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 6534.436932] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 6536.642243] Lustre: Permanently deactivating lustre-MDT0001 [ 6536.656486] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 6536.665427] Lustre: Skipped 2 previous similar messages [ 6546.511188] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 6555.106022] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6555.114057] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6555.124902] Lustre: Skipped 1 previous similar message [ 6560.230428] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6560.238907] Lustre: Skipped 5 previous similar messages [ 6560.772794] Lustre: server umount lustre-MDT0000 complete [ 6564.345147] LustreError: 239407:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788851744 with bad export cookie 16589809716926159349 [ 6564.354769] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6564.717723] Lustre: server umount lustre-MDT0001 complete [ 6570.337632] Lustre: server umount lustre-OST0000 complete [ 6580.968274] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 03:15:59 (1788851759) [ 6590.087889] Lustre: server umount lustre-OST0001 complete [ 6595.316947] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_hostid [ 6604.317800] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 6639.144586] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6639.384665] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6639.391865] Lustre: Skipped 2 previous similar messages [ 6639.407572] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6639.464112] Lustre: lustre-MDT0000: new disk, initializing [ 6639.523707] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6639.539872] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6643.327216] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 6652.607918] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6652.712530] Lustre: 247732:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6652.750990] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6652.754517] Lustre: Skipped 1 previous similar message [ 6652.818965] Lustre: lustre-MDT0001: new disk, initializing [ 6652.928615] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6652.965584] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6652.980559] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6656.715551] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 6665.939759] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6671.263232] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6676.592578] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6676.799659] Lustre: lustre-OST0000: new disk, initializing [ 6676.804238] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6676.815098] Lustre: 249701:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6676.886197] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6681.907697] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 6683.194288] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6683.212378] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6683.283662] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6689.668637] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6691.254817] LustreError: 247739:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 6699.411256] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6699.618193] Lustre: lustre-OST0001: new disk, initializing [ 6699.624601] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6699.632030] Lustre: 250816:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6701.352094] LustreError: 247739:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 6701.375375] LustreError: 247738:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 6705.298212] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 6708.961949] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6708.977635] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6709.125374] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6711.392384] LustreError: 247738:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 6715.171619] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6716.811082] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6719.462870] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6719.468067] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6719.472269] LustreError: Skipped 1 previous similar message [ 6719.479092] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6719.495607] Lustre: Skipped 3 previous similar messages [ 6719.530660] Lustre: Skipped 1 previous similar message [ 6722.289121] Lustre: lustre-OST0001: Not available for connect from 192.168.203.4@tcp (stopping) [ 6724.718527] Lustre: server umount lustre-OST0001 complete [ 6727.414757] LustreError: 249686:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.4@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6727.447060] LustreError: 249686:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6739.941555] LustreError: 249685:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6739.951094] LustreError: 249685:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 6745.060967] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6745.062853] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6745.070837] LustreError: Skipped 1 previous similar message [ 6745.071542] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6745.071546] Lustre: Skipped 2 previous similar messages [ 6745.080221] Lustre: Skipped 1 previous similar message [ 6746.609859] Lustre: server umount lustre-OST0000 complete [ 6755.301605] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6755.305226] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6755.324789] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6755.324796] Lustre: Skipped 1 previous similar message [ 6755.373491] Lustre: Skipped 1 previous similar message [ 6756.769305] Lustre: server umount lustre-MDT0000 complete [ 6760.418215] LustreError: 247744:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6760.463833] LustreError: 247744:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 6762.799138] LustreError: 250815:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788851942 with bad export cookie 16589809716926162163 [ 6762.803075] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6762.815866] LustreError: 250815:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 6777.312850] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6777.596407] Lustre: server umount lustre-MDT0001 complete [ 6790.083281] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 6793.327713] Key type lgssc unregistered [ 6793.611640] LNet: 252849:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6793.615646] LNetError: 252849:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6793.637037] LNet: Removed LNI 192.168.203.104@tcp [ 6794.425462] Key type .llcrypt unregistered [ 6794.430166] Key type ._llcrypt unregistered [ 6838.558443] Key type ._llcrypt registered [ 6838.559820] Key type .llcrypt registered [ 6838.671078] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 6840.031348] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6840.061076] alg: No test for adler32 (adler32-zlib) [ 6841.071248] Lustre: Lustre: Build Version: 2.17.57_108_g4a8ba26 [ 6841.260867] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 6842.888156] Key type lgssc registered [ 6843.870375] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6850.868543] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6850.902419] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6852.265593] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6852.282598] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6852.408975] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6855.978298] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 6863.270626] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6863.347918] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6863.367597] Lustre: 255055:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6863.548792] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6866.583423] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 6873.438735] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6880.834760] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6887.551558] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6887.809400] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6888.055380] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6894.087090] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 6894.166077] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 6900.702386] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6955.494177] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6955.495695] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6955.504448] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6955.514791] Lustre: Skipped 1 previous similar message [ 6955.530110] Lustre: Skipped 1 previous similar message [ 6959.455755] Lustre: server umount lustre-OST0000 complete [ 6963.486730] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 6975.081826] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6975.391963] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6976.428067] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6976.450327] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6976.461276] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6976.495036] Lustre: Skipped 1 previous similar message [ 6981.970339] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 6989.170647] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7027.685510] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7027.707758] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7027.719425] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7027.734685] Lustre: Skipped 1 previous similar message [ 7032.804576] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7032.821636] Lustre: Skipped 1 previous similar message [ 7033.485394] Lustre: server umount lustre-OST0000 complete [ 7037.921495] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7037.922833] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7037.938531] LustreError: Skipped 1 previous similar message [ 7037.941289] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7037.955888] Lustre: Skipped 2 previous similar messages [ 7043.045953] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7043.057270] Lustre: Skipped 2 previous similar messages [ 7051.232135] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7051.420097] Lustre: server umount lustre-MDT0000 complete [ 7054.949079] LustreError: 255049:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788852234 with bad export cookie 17851158467728295030 [ 7054.950777] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7054.976740] LustreError: 255049:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7055.287406] Lustre: server umount lustre-MDT0001 complete [ 7068.583476] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 7071.293472] Key type lgssc unregistered [ 7071.716466] LNet: 260481:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7071.723922] LNetError: 260481:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7071.743934] LNet: Removed LNI 192.168.203.104@tcp [ 7072.881310] Key type .llcrypt unregistered [ 7072.883420] Key type ._llcrypt unregistered [ 7098.912144] Key type ._llcrypt registered [ 7098.915300] Key type .llcrypt registered [ 7099.010492] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 7100.044077] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7100.183712] alg: No test for adler32 (adler32-zlib) [ 7101.506940] Lustre: Lustre: Build Version: 2.17.57_108_g4a8ba26 [ 7101.941990] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 7103.609312] Key type lgssc registered [ 7105.481988] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7116.202666] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7116.215455] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7117.793392] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7124.056865] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 7127.028928] LustreError: 261896:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7127.091509] LustreError: 261896:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7132.130166] LustreError: 261897:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7135.859762] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7136.495246] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7141.615513] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 7148.078218] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7153.044862] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7158.367149] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7158.797155] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7162.923828] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 7163.952566] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 7171.934587] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7197.666164] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7197.675763] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7197.686064] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7198.690689] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7198.694214] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7198.695069] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7202.864522] Lustre: server umount lustre-OST0000 complete [ 7208.418133] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7208.427077] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7208.437366] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7208.442278] Lustre: Skipped 1 previous similar message [ 7212.819786] Lustre: server umount lustre-MDT0000 complete [ 7214.057711] LustreError: 261891:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7214.090657] LustreError: 261891:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7216.466236] LustreError: 261878:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788852396 with bad export cookie 3305966639040910119 [ 7216.472763] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7216.481590] LustreError: 261878:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7216.962901] Lustre: server umount lustre-MDT0001 complete [ 7226.426793] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 7229.049854] Key type lgssc unregistered [ 7229.396921] LNet: 266203:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7229.406492] LNetError: 266203:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7229.430138] LNet: Removed LNI 192.168.203.104@tcp [ 7230.346638] Key type .llcrypt unregistered [ 7230.351141] Key type ._llcrypt unregistered [ 7246.031272] Key type ._llcrypt registered [ 7246.035361] Key type .llcrypt registered [ 7246.139471] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 7247.025580] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7247.075364] alg: No test for adler32 (adler32-zlib) [ 7248.162356] Lustre: Lustre: Build Version: 2.17.57_108_g4a8ba26 [ 7248.393224] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 7250.072382] Key type lgssc registered [ 7251.268830] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7259.843480] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7259.855308] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7261.665190] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7268.445930] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 7271.907262] LustreError: 267423:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7271.918290] LustreError: 267423:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7277.028940] LustreError: 267422:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7277.758856] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7278.079163] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7282.019306] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 7288.892778] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7293.878686] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7301.483154] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7301.936336] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7306.117553] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 7308.345800] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 7315.290828] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7326.691335] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7326.695293] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7326.696949] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7326.702170] Lustre: Skipped 1 previous similar message [ 7326.711021] Lustre: Skipped 1 previous similar message [ 7330.446760] Lustre: server umount lustre-OST0000 complete [ 7334.370091] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7334.375115] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7334.385918] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7336.936152] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7336.941426] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7336.959156] Lustre: Skipped 1 previous similar message [ 7340.250140] Lustre: server umount lustre-MDT0000 complete [ 7342.049548] LustreError: 267422:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7342.057582] LustreError: 267422:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7343.293428] LustreError: 267402:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788852523 with bad export cookie 3895954090294943160 [ 7343.301910] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7343.310417] LustreError: 267402:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7343.645607] Lustre: server umount lustre-MDT0001 complete [ 7355.506816] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 7358.780284] Key type lgssc unregistered [ 7359.098713] LNet: 271097:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7359.115665] LNetError: 271097:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7359.199091] LNet: Removed LNI 192.168.203.104@tcp [ 7360.033769] Key type .llcrypt unregistered [ 7360.040190] Key type ._llcrypt unregistered [ 7394.949747] Key type ._llcrypt registered [ 7394.952952] Key type .llcrypt registered [ 7395.134523] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 7396.474421] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7396.534659] alg: No test for adler32 (adler32-zlib) [ 7397.568251] Lustre: Lustre: Build Version: 2.17.57_108_g4a8ba26 [ 7397.768235] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 7399.497523] Key type lgssc registered [ 7400.623312] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7408.672741] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7408.691025] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7410.193055] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7415.222994] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 7420.442372] LustreError: 272514:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7420.466832] LustreError: 272514:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7425.513899] LustreError: 272513:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7427.234862] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7428.039129] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7433.235406] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 7440.127493] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7445.008448] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7450.853482] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7451.086843] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7455.218836] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7455.802095] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 7462.913371] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7487.973479] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7487.979824] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7487.992341] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7491.054147] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7491.066553] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7491.078286] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7491.121899] Lustre: Skipped 1 previous similar message [ 7493.353509] Lustre: server umount lustre-OST0000 complete [ 7499.747874] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7499.761319] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7499.783474] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7503.618579] Lustre: server umount lustre-MDT0000 complete [ 7506.406843] LustreError: 272513:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7506.426915] LustreError: 272513:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7508.871042] LustreError: 272495:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788852688 with bad export cookie 18226373705818444651 [ 7508.888560] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7508.890258] LustreError: 272495:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7509.471393] Lustre: server umount lustre-MDT0001 complete [ 7521.930703] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 7524.930861] Key type lgssc unregistered [ 7525.334780] LNet: 276820:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7525.351407] LNetError: 276820:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7525.364077] LNet: Removed LNI 192.168.203.104@tcp [ 7526.298792] Key type .llcrypt unregistered [ 7526.310929] Key type ._llcrypt unregistered [ 7543.051023] Key type ._llcrypt registered [ 7543.053056] Key type .llcrypt registered [ 7543.147526] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 7543.941804] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7544.007325] alg: No test for adler32 (adler32-zlib) [ 7545.015704] Lustre: Lustre: Build Version: 2.17.57_108_g4a8ba26 [ 7545.271968] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 7546.928486] Key type lgssc registered [ 7548.087719] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7555.750326] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7555.770080] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7557.487163] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7561.880734] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 7567.843498] LustreError: 278052:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7567.863155] LustreError: 278052:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7572.227849] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7572.718078] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7578.299723] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 7584.697172] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7590.212873] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7596.289189] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7596.688167] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7602.438877] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 7602.812625] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 7609.607816] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7623.654796] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7623.656643] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7623.665321] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7623.665736] LustreError: Skipped 1 previous similar message [ 7623.689619] Lustre: Skipped 1 previous similar message [ 7625.625675] Lustre: server umount lustre-OST0000 complete [ 7628.781145] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7628.781549] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7628.789177] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7628.798420] Lustre: Skipped 1 previous similar message [ 7628.812636] Lustre: Skipped 2 previous similar messages [ 7633.899833] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7633.915472] Lustre: Skipped 1 previous similar message [ 7635.019391] Lustre: server umount lustre-MDT0000 complete [ 7638.554420] LustreError: 278019:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788852818 with bad export cookie 7965971501076084857 [ 7638.559539] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7638.564523] LustreError: 278019:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7638.980746] Lustre: server umount lustre-MDT0001 complete [ 7648.507576] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 7651.084764] Key type lgssc unregistered [ 7651.374896] LNet: 281751:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7651.387513] LNetError: 281751:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7652.455917] LNet: Removed LNI 192.168.203.104@tcp [ 7653.264773] Key type .llcrypt unregistered [ 7653.273438] Key type ._llcrypt unregistered [ 7665.511433] Key type ._llcrypt registered [ 7665.514120] Key type .llcrypt registered [ 7665.599684] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 7681.566383] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_hostid [ 7692.374272] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 7692.958901] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7692.981732] alg: No test for adler32 (adler32-zlib) [ 7694.076806] Lustre: Lustre: Build Version: 2.17.57_108_g4a8ba26 [ 7694.322958] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 7695.976181] Key type lgssc registered [ 7696.920410] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7734.198409] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7734.284646] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7735.683703] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7735.732697] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7735.850272] Lustre: lustre-MDT0000: new disk, initializing [ 7736.040740] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7736.105079] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7740.694802] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 7756.279588] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7756.498105] Lustre: 285279:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7756.565947] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7756.573181] Lustre: Skipped 1 previous similar message [ 7756.637454] Lustre: lustre-MDT0001: new disk, initializing [ 7756.726775] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7756.780937] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7756.799934] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7762.075856] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 7771.277224] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7777.268703] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7783.570507] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7783.787582] Lustre: lustre-OST0000: new disk, initializing [ 7783.790793] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7783.795279] Lustre: 287252:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7783.854808] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7789.899744] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 7790.724302] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7790.757164] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7790.874485] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7799.655676] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7806.437777] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7806.716366] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7813.243409] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7813.504753] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7816.690670] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7816.690942] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7816.720646] Lustre: Skipped 1 previous similar message [ 7816.724331] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7821.890518] Lustre: server umount lustre-OST0000 complete [ 7826.913800] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7826.929139] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7826.931516] Lustre: Skipped 2 previous similar messages [ 7831.841735] Lustre: server umount lustre-MDT0000 complete [ 7836.217284] LustreError: 287251:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788853016 with bad export cookie 5033169357804970111 [ 7836.222873] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7836.232075] LustreError: 287251:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7836.974740] Lustre: server umount lustre-MDT0001 complete [ 7846.992918] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 03:37:04 (1788853024) [ 7854.431738] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7854.789593] LustreError: 289664:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7854.816242] LustreError: 289664:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 7854.863122] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7859.217381] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 7860.197703] LustreError: 289665:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7864.293421] LustreError: 289664:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7866.383432] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7866.780491] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7870.300773] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 7876.089600] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7881.961581] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7888.661432] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7888.997980] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7895.014267] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 7902.888816] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7911.910169] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7911.924812] LustreError: Skipped 1 previous similar message [ 7911.931536] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7911.953750] Lustre: Skipped 1 previous similar message [ 7911.960149] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7914.981608] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7914.991072] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7914.997955] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7916.983119] Lustre: server umount lustre-OST0000 complete [ 7919.814530] Lustre: server umount lustre-MDT0000 complete [ 7920.098482] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7920.102768] LustreError: 289661:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7920.109360] Lustre: Skipped 1 previous similar message [ 7920.118818] LustreError: 289661:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7922.914457] LustreError: 290399:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788853102 with bad export cookie 5033169357804970993 [ 7922.918535] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7922.920740] LustreError: 290399:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7923.186593] Lustre: server umount lustre-MDT0001 complete [ 7932.407230] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 7935.178199] Key type lgssc unregistered [ 7935.442952] LNet: 293195:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7935.451801] LNetError: 293195:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7935.470779] LNet: Removed LNI 192.168.203.104@tcp [ 7936.157797] Key type .llcrypt unregistered [ 7936.163388] Key type ._llcrypt unregistered [ 7964.553691] Key type ._llcrypt registered [ 7964.558714] Key type .llcrypt registered [ 7964.719229] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_hostid [ 7980.543490] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 7981.600819] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7981.783244] alg: No test for adler32 (adler32-zlib) [ 7982.928912] Lustre: Lustre: Build Version: 2.17.57_108_g4a8ba26 [ 7983.306062] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 7985.000172] Key type lgssc registered [ 7986.075888] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8031.874529] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8031.892600] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8033.251432] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8033.330056] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8033.497700] Lustre: lustre-MDT0000: new disk, initializing [ 8033.635977] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8033.660827] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8039.301394] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 8050.501715] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8050.586595] Lustre: 297018:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8050.680293] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8050.683826] Lustre: Skipped 1 previous similar message [ 8050.742950] Lustre: lustre-MDT0001: new disk, initializing [ 8050.814932] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8050.870181] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8050.885498] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8056.561482] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 8065.598756] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8070.982469] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8078.539187] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8078.747141] Lustre: lustre-OST2710: new disk, initializing [ 8078.752117] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 8078.757765] Lustre: 298988:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8078.871849] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 8085.958890] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 8087.106806] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 8087.116389] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 8087.212286] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 8096.058737] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8102.126579] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8102.269122] Lustre: lustre-OST03e8: new disk, initializing [ 8102.273244] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 8102.278525] Lustre: 300051:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8102.374487] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 8107.151277] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 8110.681807] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 8110.692580] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 8110.787746] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 8115.634556] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8124.087615] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 8124.374695] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8129.361933] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 8129.698663] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8134.311814] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 8134.593993] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8140.499070] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 8140.820660] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8146.937100] Lustre: lustre-MDT0000-lwp-OST03e8: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8146.948430] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8146.950290] Lustre: Skipped 1 previous similar message [ 8146.964561] Lustre: Skipped 3 previous similar messages [ 8148.420488] Lustre: server umount lustre-MDT0000 complete [ 8152.348197] LustreError: 300050:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788853332 with bad export cookie 11661022277925687278 [ 8152.357488] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8152.367435] LustreError: 300050:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8152.817458] Lustre: server umount lustre-MDT0001 complete [ 8166.566261] Lustre: server umount lustre-OST2710 complete [ 8181.260876] Lustre: server umount lustre-OST03e8 complete [ 8198.641128] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_hostid [ 8208.744720] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 8247.586259] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8247.800956] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8247.824917] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8247.910090] Lustre: lustre-MDT0000: new disk, initializing [ 8247.992168] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8248.030274] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8252.977576] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 8266.982019] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8267.239732] Lustre: 305301:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8267.476153] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8267.500734] Lustre: Skipped 1 previous similar message [ 8267.733537] Lustre: lustre-MDT0001: new disk, initializing [ 8267.921344] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8267.983574] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8268.004111] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8273.989103] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 8283.835811] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8289.508844] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8295.571324] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8295.920698] Lustre: lustre-OST0000: new disk, initializing [ 8295.923691] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8295.928858] Lustre: 307270:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8296.013231] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8302.006904] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 8305.730114] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8305.735753] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8305.805883] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8310.943189] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8315.841174] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8315.994593] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8320.477859] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8320.918072] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8326.628909] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8326.629469] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8326.635382] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8326.637595] Lustre: Skipped 3 previous similar messages [ 8326.654402] LustreError: Skipped 1 previous similar message [ 8329.755622] Lustre: server umount lustre-OST0000 complete [ 8334.323100] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8334.337243] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8334.373669] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8334.390823] Lustre: Skipped 1 previous similar message [ 8336.865614] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8336.867312] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8339.423579] Lustre: server umount lustre-MDT0000 complete [ 8341.989171] LustreError: 305313:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8342.003770] LustreError: 305313:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 8342.785151] LustreError: 307268:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788853522 with bad export cookie 11661022277925688958 [ 8342.785722] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8342.795666] LustreError: 307268:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8343.087544] Lustre: server umount lustre-MDT0001 complete [ 8351.102406] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 03:45:29 (1788853529) [ 8353.211088] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 8355.005762] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 03:45:33 (1788853533) [ 8358.872397] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 8387.142834] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8387.495444] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8387.536788] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8387.677721] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8392.942338] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 8402.142532] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8402.382514] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8402.432831] Lustre: 311128:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8409.149165] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 8430.337798] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8430.557459] Lustre: Denying initial registration attempt from nid 192.168.203.104@tcp, specified as failover [ 8430.569419] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 8430.578675] LustreError: 312581:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -99 [ 8430.585477] LustreError: 312581:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 8430.588088] LustreError: 312581:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 8430.633375] Lustre: server umount lustre-OST0000 complete [ 8430.635384] LustreError: 312581:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 8436.712522] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8436.736913] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8436.742955] Lustre: Skipped 1 previous similar message [ 8436.752300] Lustre: Skipped 2 previous similar messages [ 8443.094068] Lustre: server umount lustre-MDT0000 complete [ 8446.948490] LustreError: 311140:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8446.977686] LustreError: 311140:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8448.811559] LustreError: 311119:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788853628 with bad export cookie 11661022277925689840 [ 8448.829814] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8448.844215] LustreError: 311119:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8449.397859] Lustre: server umount lustre-MDT0001 complete [ 8461.623888] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 8465.118415] Key type lgssc unregistered [ 8465.495662] LNet: 313703:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8465.511126] LNetError: 313703:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8465.545537] LNet: Removed LNI 192.168.203.104@tcp [ 8466.957730] Key type .llcrypt unregistered [ 8466.959636] Key type ._llcrypt unregistered [ 8481.588471] Key type ._llcrypt registered [ 8481.591164] Key type .llcrypt registered [ 8481.670763] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 8482.566456] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8482.597400] alg: No test for adler32 (adler32-zlib) [ 8483.782909] Lustre: Lustre: Build Version: 2.17.57_108_g4a8ba26 [ 8484.056016] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 8485.792164] Key type lgssc registered [ 8487.288203] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8536.701157] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 8545.326477] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8545.363042] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8546.634092] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8546.653639] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8546.866656] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8550.827216] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 8563.025262] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8563.126684] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8563.177230] Lustre: 316462:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8563.626524] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8569.690330] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 8579.557385] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8588.587562] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8611.315823] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8611.494989] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8611.738321] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8618.176838] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 8626.830582] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8632.292601] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8632.307347] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8632.332780] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8632.344831] Lustre: Skipped 1 previous similar message [ 8636.275723] Lustre: server umount lustre-OST0000 complete [ 8645.620047] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8645.707301] LustreError: Skipped 1 previous similar message [ 8645.719343] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8645.737963] Lustre: Skipped 1 previous similar message [ 8645.781904] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8647.655491] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8647.657167] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8647.690538] Lustre: Skipped 1 previous similar message [ 8652.772506] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8652.796335] Lustre: Skipped 1 previous similar message [ 8657.376116] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8657.506436] Lustre: server umount lustre-MDT0000 complete [ 8660.440365] LustreError: 316455:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788853840 with bad export cookie 4756740899902168582 [ 8660.453936] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8660.702658] Lustre: server umount lustre-MDT0001 complete [ 8669.198769] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 8672.127572] Key type lgssc unregistered [ 8672.481580] LNet: 320039:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8672.499077] LNetError: 320039:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8672.530265] LNet: Removed LNI 192.168.203.104@tcp [ 8673.351728] Key type .llcrypt unregistered [ 8673.354634] Key type ._llcrypt unregistered [ 8696.671127] Key type ._llcrypt registered [ 8696.673543] Key type .llcrypt registered [ 8696.729941] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 8697.694037] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8697.702598] alg: No test for adler32 (adler32-zlib) [ 8698.710693] Lustre: Lustre: Build Version: 2.17.57_108_g4a8ba26 [ 8698.982849] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 8700.664165] Key type lgssc registered [ 8701.753778] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8710.098841] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8710.133402] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8711.881174] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8717.804871] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 8721.384635] LustreError: 321454:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8721.400510] LustreError: 321454:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8726.497359] LustreError: 321455:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8727.774777] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8728.060421] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8732.354864] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 8738.780077] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8743.894707] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8751.814788] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8752.422390] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8759.084371] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 8766.825490] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8777.187798] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8777.193480] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8777.200762] Lustre: Skipped 1 previous similar message [ 8777.216533] Lustre: Skipped 2 previous similar messages [ 8778.563657] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.4@tcp (stopping) [ 8781.483546] Lustre: server umount lustre-MDT0000 complete [ 8782.313806] LustreError: 321454:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8782.358257] LustreError: 321454:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8786.471314] LustreError: 321436:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788853966 with bad export cookie 16160583903856130410 [ 8786.484762] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8786.493862] LustreError: 321436:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8787.427880] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8787.429306] LustreError: 321451:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8787.449190] Lustre: Skipped 1 previous similar message [ 8787.451326] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8787.492304] LustreError: 321451:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8791.457237] LustreError: 321449:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8791.457615] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8791.469358] LustreError: 321449:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8791.479791] Lustre: Skipped 1 previous similar message [ 8793.092904] Lustre: server umount lustre-MDT0001 complete [ 8799.884584] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 8809.771368] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8811.936138] LustreError: 325035:0:(import.c:339:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [ 8811.949946] LustreError: 325035:0:(import.c:363:ptlrpc_invalidate_import()) @@@ still on sending list req@0000000049976897 x1875749244010368/t0(0) o250->MGC192.168.203.104@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1788853991 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8811.989602] LustreError: 325035:0:(import.c:373:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. [ 8812.011899] LustreError: 320763:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000b52d9759 x1875749244010752/t0(0) o250->MGC192.168.203.104@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8812.735660] LustreError: 323287:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8812.873637] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8812.988394] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 8820.997331] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 8821.041150] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 8831.474129] LustreError: 325048:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8831.499569] LustreError: 325048:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 8832.671874] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8833.196657] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8837.532038] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 8838.638444] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 8846.621637] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8848.295174] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8854.222357] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8856.670497] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8863.713703] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8863.737600] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8863.769234] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8864.232873] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8874.496954] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8874.520819] Lustre: Skipped 5 previous similar messages [ 8876.001138] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8876.185977] Lustre: server umount lustre-OST0000 complete [ 8884.203489] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8884.213920] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8884.233049] Lustre: Skipped 1 previous similar message [ 8886.213779] Lustre: server umount lustre-MDT0000 complete [ 8889.832302] LustreError: 325053:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8889.864414] LustreError: 325053:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8891.145494] LustreError: 321434:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788854071 with bad export cookie 16160583903856137550 [ 8891.149431] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8891.154020] LustreError: 321434:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8891.356670] Lustre: server umount lustre-MDT0001 complete [ 8904.613688] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 8907.919368] Key type lgssc unregistered [ 8908.192852] LNet: 327725:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8908.200379] LNetError: 327725:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8908.217604] LNet: Removed LNI 192.168.203.104@tcp [ 8909.127853] Key type .llcrypt unregistered [ 8909.134391] Key type ._llcrypt unregistered [ 8933.061188] Key type ._llcrypt registered [ 8933.064739] Key type .llcrypt registered [ 8933.156417] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 8934.105837] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8934.191294] alg: No test for adler32 (adler32-zlib) [ 8935.510460] Lustre: Lustre: Build Version: 2.17.57_108_g4a8ba26 [ 8935.751548] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 8937.441494] Key type lgssc registered [ 8939.133215] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8946.999857] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8947.014847] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8948.544525] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8953.469640] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 8958.947177] LustreError: 329136:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8958.967291] LustreError: 329136:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8961.569253] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8961.989701] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8965.969373] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 8972.879621] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8978.302245] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8982.497657] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8982.508319] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8982.540387] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8983.521031] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8983.542443] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8986.673032] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8986.684855] Lustre: Skipped 2 previous similar messages [ 8988.693024] Lustre: server umount lustre-MDT0000 complete [ 8991.715701] LustreError: 329862:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8991.727061] LustreError: 329862:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8992.784177] LustreError: 329117:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788854172 with bad export cookie 17356675278670612907 [ 8992.801317] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8992.811190] LustreError: 329117:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8993.670517] Lustre: server umount lustre-MDT0001 complete [ 9011.979996] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9012.252732] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9012.277986] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9012.394192] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9016.162867] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 9027.161864] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9027.291605] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9027.342449] Lustre: 331722:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9027.701511] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9032.954977] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 9040.951303] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9048.266730] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9060.471087] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9060.858446] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 9060.876463] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 9060.888584] LustreError: 333609:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -2 [ 9060.892679] LustreError: 333609:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 9060.897725] LustreError: 333609:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9060.968266] Lustre: server umount lustre-OST0000 complete [ 9060.976557] LustreError: 333609:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 9072.107359] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9072.338667] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9072.488908] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9079.480372] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 9082.485467] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 9086.959972] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9094.058639] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9094.132456] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 9094.239033] Lustre: lustre-OST0001: new disk, initializing [ 9094.246063] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9094.254247] Lustre: 335152:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9094.359837] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9099.292797] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 9102.415649] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9102.429935] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9102.520916] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9106.095838] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9113.080880] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9113.085733] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9113.085907] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9113.095886] Lustre: Skipped 1 previous similar message [ 9113.115185] Lustre: Skipped 1 previous similar message [ 9118.182407] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9118.194021] Lustre: Skipped 1 previous similar message [ 9123.303749] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9123.382459] Lustre: server umount lustre-OST0001 complete [ 9128.420429] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9128.425276] LustreError: Skipped 1 previous similar message [ 9128.427719] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9128.428258] LustreError: 334075:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9128.443854] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9128.452439] LustreError: 334075:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9128.460688] Lustre: Skipped 3 previous similar messages [ 9133.540625] LustreError: 334078:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9133.552083] LustreError: 334078:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9138.663087] LustreError: 334078:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9138.685159] LustreError: 334078:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9141.216230] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9141.633055] Lustre: server umount lustre-OST0000 complete [ 9148.902886] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9148.908803] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9148.918213] Lustre: Skipped 2 previous similar messages [ 9148.926149] Lustre: Skipped 5 previous similar messages [ 9153.747728] Lustre: server umount lustre-MDT0000 complete [ 9154.034857] LustreError: 331733:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9154.055593] LustreError: 331733:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9158.057862] LustreError: 335151:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788854338 with bad export cookie 17356675278670613460 [ 9158.067014] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9158.097111] LustreError: 335151:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9158.587234] Lustre: server umount lustre-MDT0001 complete [ 9170.305440] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 9173.784463] Key type lgssc unregistered [ 9174.052770] LNet: 337113:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9174.061389] LNetError: 337113:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9174.081548] LNet: Removed LNI 192.168.203.104@tcp [ 9175.107195] Key type .llcrypt unregistered [ 9175.109084] Key type ._llcrypt unregistered [ 9258.029468] Key type ._llcrypt registered [ 9258.034631] Key type .llcrypt registered [ 9258.211146] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_hostid [ 9274.016179] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 9275.330135] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9275.357301] alg: No test for adler32 (adler32-zlib) [ 9276.440630] Lustre: Lustre: Build Version: 2.17.57_108_g4a8ba26 [ 9276.805544] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 9278.559131] Key type lgssc registered [ 9280.590867] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9326.852362] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9326.915033] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9328.202384] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9328.221388] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9328.292943] Lustre: lustre-MDT0000: new disk, initializing [ 9328.406313] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9328.420463] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9332.423826] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 9346.153750] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9346.358801] Lustre: 342061:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9346.452661] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9346.457876] Lustre: Skipped 1 previous similar message [ 9346.698857] Lustre: lustre-MDT0001: new disk, initializing [ 9347.011242] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9347.116643] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9347.131755] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9354.458242] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 9365.256925] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9370.967717] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9378.643189] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9378.831683] Lustre: lustre-OST0000: new disk, initializing [ 9378.834764] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9378.839824] Lustre: 344032:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9378.898885] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9385.915823] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 9389.155098] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9389.181536] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9389.300619] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9396.152925] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9401.267949] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9401.542910] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9406.925813] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9407.269390] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9410.018236] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9410.021150] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9410.048545] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9410.055978] Lustre: Skipped 1 previous similar message [ 9415.140098] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9415.144544] Lustre: Skipped 1 previous similar message [ 9415.469913] Lustre: server umount lustre-OST0000 complete [ 9420.265086] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9420.278712] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9420.290288] Lustre: Skipped 2 previous similar messages [ 9420.301227] Lustre: Skipped 1 previous similar message [ 9425.566224] Lustre: server umount lustre-MDT0000 complete [ 9429.967025] LustreError: 342857:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788854609 with bad export cookie 4270960188861742004 [ 9429.979934] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9429.989624] LustreError: 342857:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9430.509176] Lustre: server umount lustre-MDT0001 complete [ 9441.219225] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 04:03:39 (1788854619) [ 9450.216721] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 04:03:48 (1788854628) [ 9458.876152] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9459.311765] LustreError: 346788:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9459.325776] LustreError: 346788:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9459.406793] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9463.814352] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 9464.812997] LustreError: 346789:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9469.923414] LustreError: 346788:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9471.897148] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9472.379722] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9477.706644] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 9485.882172] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9492.969529] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9500.685596] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9501.167343] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9508.494192] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 9516.233496] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9524.524205] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 9526.523942] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 9528.176884] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 9530.111457] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 9532.213902] Lustre: Failing over lustre-MDT0000 [ 9532.577230] Lustre: server umount lustre-MDT0000 complete [ 9533.927751] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9533.929904] LustreError: 346788:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9533.946445] Lustre: Skipped 2 previous similar messages [ 9533.947115] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9533.947120] LustreError: Skipped 1 previous similar message [ 9533.994749] LustreError: 346788:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9539.045383] LustreError: 346784:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9539.070106] LustreError: 346784:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9549.290513] LustreError: 346789:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9549.314574] LustreError: 346789:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 9549.326364] Lustre: 340270:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788854713/real 1788854713] req@00000000777fb295 x1875749850124416/t0(0) o400->MGC192.168.203.104@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788854729 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9549.345234] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9551.603185] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9559.547153] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x3b457fa9caf08339 [ 9559.603298] Lustre: MGC192.168.203.104@tcp: Connection restored to 0@lo (at 0@lo) [ 9559.999321] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9560.116551] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9560.813196] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9565.169864] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9565.206163] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 9565.263793] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 9565.798934] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 9575.982823] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9577.662670] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9581.210574] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 9583.268571] Lustre: Failing over lustre-MDT0000 [ 9583.568308] Lustre: server umount lustre-MDT0000 complete [ 9585.644908] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9585.646334] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9585.662702] Lustre: Skipped 1 previous similar message [ 9585.666824] LustreError: 346785:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9585.701509] LustreError: 346785:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [ 9601.504127] Lustre: 340271:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788854765/real 1788854765] req@000000003a8feb28 x1875749850144896/t0(0) o400->MGC192.168.203.104@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788854781 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9601.531151] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9602.875031] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9611.747679] LustreError: 340267:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000003bf9d2c9 x1875749850149504/t0(0) o250->MGC192.168.203.104@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9612.046696] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9612.088836] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9613.043322] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9617.396548] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9617.400323] Lustre: Skipped 2 previous similar messages [ 9617.433756] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 9617.471703] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 9617.621030] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 9628.425925] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9630.744861] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9637.859321] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9637.861125] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9637.874368] Lustre: Skipped 2 previous similar messages [ 9637.874750] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9637.890277] Lustre: Skipped 2 previous similar messages [ 9642.978415] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9642.988123] Lustre: Skipped 2 previous similar messages [ 9644.183726] Lustre: server umount lustre-MDT0000 complete [ 9647.812138] LustreError: 346770:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788854827 with bad export cookie 4270960188861744720 [ 9647.823745] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9647.828784] LustreError: 346770:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9648.077579] Lustre: server umount lustre-MDT0001 complete [ 9662.665678] Lustre: server umount lustre-OST0000 complete [ 9678.319061] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9678.743743] LustreError: 353138:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9678.766572] LustreError: 353138:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 21 previous similar messages [ 9678.806941] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9685.142548] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 9696.122568] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9701.491605] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 9709.978146] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9716.236640] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9722.969912] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9724.413913] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 9730.922402] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 9739.969546] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9752.033758] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9752.044041] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9752.064361] Lustre: Skipped 2 previous similar messages [ 9752.065713] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9752.080628] Lustre: Skipped 1 previous similar message [ 9756.488291] Lustre: server umount lustre-OST0000 complete [ 9765.884697] Lustre: server umount lustre-MDT0000 complete [ 9767.396364] LustreError: 353134:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9767.408140] LustreError: 353134:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 9769.834522] LustreError: 353163:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788854949 with bad export cookie 4270960188861745469 [ 9769.836206] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9769.843209] LustreError: 353163:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9770.187471] Lustre: server umount lustre-MDT0001 complete [ 9781.310127] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 9784.115611] Key type lgssc unregistered [ 9784.425722] LNet: 356673:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9784.434277] LNetError: 356673:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9784.464029] LNet: Removed LNI 192.168.203.104@tcp [ 9785.589529] Key type .llcrypt unregistered [ 9785.592939] Key type ._llcrypt unregistered [ 9815.099500] Key type ._llcrypt registered [ 9815.104843] Key type .llcrypt registered [ 9815.254945] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 9816.912620] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9817.046083] alg: No test for adler32 (adler32-zlib) [ 9818.239956] Lustre: Lustre: Build Version: 2.17.57_108_g4a8ba26 [ 9818.564595] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 9820.328168] Key type lgssc registered [ 9821.605862] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9830.534167] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9830.556078] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9832.156541] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9836.960872] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 9842.659254] LustreError: 358190:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9842.679427] LustreError: 358190:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9846.221554] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9846.705054] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9851.418272] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 9858.934204] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9864.849979] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9871.242045] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9871.330942] LustreError: 360011:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9871.525756] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9877.398540] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 9879.028380] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 9885.374338] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9903.079361] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9903.093928] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9903.123362] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9904.613039] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9904.614742] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9904.633982] Lustre: Skipped 1 previous similar message [ 9904.671526] Lustre: Skipped 1 previous similar message [ 9909.729870] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9909.744944] Lustre: Skipped 3 previous similar messages [ 9914.336142] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9914.549847] Lustre: server umount lustre-MDT0000 complete [ 9914.851056] LustreError: 358191:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9914.878270] LustreError: 358191:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9918.290688] LustreError: 358171:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788855098 with bad export cookie 6687532779254617743 [ 9918.295719] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9918.307196] LustreError: 358171:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9918.654399] Lustre: server umount lustre-MDT0001 complete [ 9932.742706] Lustre: server umount lustre-OST0000 complete [ 9947.347000] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9947.737730] LustreError: 361923:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9947.806879] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9953.253463] LustreError: 361924:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9953.680862] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 9962.479087] LustreError: 361923:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9962.504303] LustreError: 361923:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9964.738078] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9965.401738] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9971.325117] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 9979.188185] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9985.085764] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9991.078841] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9991.508172] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9992.569190] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 9998.121684] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [10005.212559] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10016.226561] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10016.234827] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10016.252713] Lustre: Skipped 1 previous similar message [10016.266898] Lustre: Skipped 2 previous similar messages [10021.348096] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10021.356294] Lustre: Skipped 2 previous similar messages [10022.205256] Lustre: server umount lustre-MDT0000 complete [10025.138193] LustreError: 361903:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788855205 with bad export cookie 6687532779254618856 [10025.150347] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10025.162171] LustreError: 361903:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10025.408991] Lustre: server umount lustre-MDT0001 complete [10039.692068] Lustre: server umount lustre-OST0000 complete [10055.739164] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10056.298080] LustreError: 365836:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10056.412026] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10061.747406] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [10071.777360] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10072.035054] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [10079.510101] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [10088.543151] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10095.568069] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10102.757525] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10103.341886] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10103.354357] Lustre: Skipped 1 previous similar message [10111.439484] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [10119.966665] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10126.894061] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [10137.062285] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10137.063818] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10137.066485] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10137.087533] Lustre: Skipped 2 previous similar messages [10149.856108] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10150.124844] Lustre: server umount lustre-OST0000 complete [10154.465558] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10154.478785] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10168.288270] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10168.510740] Lustre: server umount lustre-MDT0000 complete [10172.912303] LustreError: 368360:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10172.932840] LustreError: 368360:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10172.946181] LustreError: 368642:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788855352 with bad export cookie 6687532779254619871 [10172.962435] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10173.321710] Lustre: server umount lustre-MDT0001 complete [10184.487388] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [10187.577443] Key type lgssc unregistered [10187.976519] LNet: 369395:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10187.981738] LNetError: 369395:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10187.999663] LNet: Removed LNI 192.168.203.104@tcp [10188.703779] Key type .llcrypt unregistered [10188.705922] Key type ._llcrypt unregistered [10217.076825] Key type ._llcrypt registered [10217.080419] Key type .llcrypt registered [10217.175218] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [10218.268283] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10218.289835] alg: No test for adler32 (adler32-zlib) [10219.371916] Lustre: Lustre: Build Version: 2.17.57_108_g4a8ba26 [10219.637435] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [10221.307205] Key type lgssc registered [10222.364193] Lustre: Echo OBD driver; http://www.lustre.org/ [10229.852066] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10229.873337] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10229.878682] LustreError: 370833:0:(osd_handler.c:8547:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [10229.916828] LustreError: 370833:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [10229.932993] LustreError: 370833:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [10229.943142] LustreError: 370833:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [10229.959635] LustreError: 370833:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [10237.532142] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [10237.545689] LustreError: 371231:0:(osd_handler.c:8547:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [10237.567884] LustreError: 371231:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [10237.572033] LustreError: 371231:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [10237.576214] LustreError: 371231:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [10237.584044] LustreError: 371231:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [10259.851717] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [10263.441910] Key type lgssc unregistered [10263.893903] LNet: 372233:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10263.899922] LNetError: 372233:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10263.932780] LNet: Removed LNI 192.168.203.104@tcp [10265.134542] Key type .llcrypt unregistered [10265.141606] Key type ._llcrypt unregistered [10284.345597] Key type ._llcrypt registered [10284.355527] Key type .llcrypt registered [10284.505549] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_hostid [10299.964365] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [10301.063779] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10301.158825] alg: No test for adler32 (adler32-zlib) [10302.217745] Lustre: Lustre: Build Version: 2.17.57_108_g4a8ba26 [10302.474353] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [10304.168223] Key type lgssc registered [10305.290941] Lustre: Echo OBD driver; http://www.lustre.org/ [10340.034457] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10340.066154] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10341.381772] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10341.417858] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10341.550836] Lustre: lustre-MDT0000: new disk, initializing [10341.657815] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10341.685653] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10346.365597] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [10356.645419] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10356.818297] Lustre: 375387:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10356.900170] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10356.904817] Lustre: Skipped 1 previous similar message [10356.984308] Lustre: lustre-MDT0001: new disk, initializing [10357.145966] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10357.226371] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10357.237039] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10361.732968] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [10371.699469] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10378.126751] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10385.845284] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10386.204887] Lustre: lustre-OST0000: new disk, initializing [10386.217654] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10386.233970] Lustre: 377355:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10386.349247] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10393.353634] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [10395.232803] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10395.253148] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10395.426218] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10403.583702] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10408.506014] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10408.734154] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10414.428073] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10414.687963] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10421.221375] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10421.222936] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10421.227224] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10421.233963] Lustre: Skipped 1 previous similar message [10422.904645] Lustre: server umount lustre-OST0000 complete [10428.902482] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10428.916661] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10428.931834] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10428.940924] Lustre: Skipped 1 previous similar message [10431.462285] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10431.462883] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10432.917532] Lustre: server umount lustre-MDT0000 complete [10436.173808] LustreError: 377354:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788855616 with bad export cookie 944305123575745294 [10436.177926] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10436.190384] LustreError: 377354:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10436.524852] Lustre: server umount lustre-MDT0001 complete [10445.753635] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [10447.679682] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 04:20:25 (1788855625) [10456.276758] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10456.825861] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10462.180261] LustreError: 379816:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10462.205645] LustreError: 379816:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10462.657413] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [10466.276841] LustreError: 379817:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10471.403756] LustreError: 379816:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10475.016816] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10475.498778] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10480.698528] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [10489.126586] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10494.746720] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10501.712617] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10502.127564] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10511.011227] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [10521.333175] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10527.062428] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10527.470115] Lustre: lustre-OST0001: new disk, initializing [10527.472782] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10527.493980] Lustre: 382654:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10527.773646] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10534.556946] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10534.569372] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10534.719054] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10534.948774] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [10544.655976] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10550.243044] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10550.267690] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10550.285721] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10550.293277] Lustre: Skipped 2 previous similar messages [10554.009531] Lustre: server umount lustre-OST0001 complete [10555.382638] LustreError: 381639:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10555.400057] LustreError: 381639:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10560.494266] LustreError: 381638:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10560.526759] LustreError: 381638:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10570.729850] LustreError: 382906:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10570.743622] LustreError: 382906:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10571.745904] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10571.750443] LustreError: Skipped 1 previous similar message [10571.754200] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10571.762700] Lustre: Skipped 1 previous similar message [10571.766146] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10571.769402] Lustre: Skipped 1 previous similar message [10577.309702] Lustre: server umount lustre-OST0000 complete [10583.008987] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10583.024689] LustreError: Skipped 1 previous similar message [10583.040341] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10583.073854] Lustre: Skipped 1 previous similar message [10583.080214] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10583.085119] Lustre: Skipped 3 previous similar messages [10587.966233] Lustre: server umount lustre-MDT0000 complete [10591.202495] LustreError: 379817:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10591.217465] LustreError: 379817:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10592.000411] LustreError: 379796:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788855771 with bad export cookie 944305123575746176 [10592.002550] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10592.029800] LustreError: 379796:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10598.306451] Lustre: server umount lustre-MDT0001 complete [10611.288426] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [10614.680869] Key type lgssc unregistered [10615.206536] LNet: 384640:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10615.228202] LNetError: 384640:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10615.273573] LNet: Removed LNI 192.168.203.104@tcp [10616.286225] Key type .llcrypt unregistered [10616.288196] Key type ._llcrypt unregistered [10668.334651] Key type ._llcrypt registered [10668.338311] Key type .llcrypt registered [10668.439445] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [10669.155536] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10669.226087] alg: No test for adler32 (adler32-zlib) [10670.205960] Lustre: Lustre: Build Version: 2.17.57_108_g4a8ba26 [10670.464098] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [10672.201246] Key type lgssc registered [10673.431798] Lustre: Echo OBD driver; http://www.lustre.org/ [10680.853371] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10680.875361] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10682.113935] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10682.145096] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10682.269363] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10688.090404] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [10698.002027] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10698.162437] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10698.188457] Lustre: 387436:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10698.503781] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10702.864306] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [10709.857493] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10715.552990] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10719.201952] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10719.205866] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10719.212292] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10719.264761] Lustre: Skipped 1 previous similar message [10723.876578] Lustre: server umount lustre-MDT0000 complete [10724.327131] LustreError: 387448:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10724.353134] LustreError: 387448:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10727.251147] LustreError: 388193:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788855907 with bad export cookie 1882548798007397500 [10727.259648] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10727.276904] LustreError: 388193:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10727.505736] Lustre: server umount lustre-MDT0001 complete [10737.117532] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 04:25:15 (1788855915) [10743.541264] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10744.013432] LustreError: 389947:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10744.112227] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10748.910270] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [10749.423659] LustreError: 389948:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10754.536295] LustreError: 389947:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10756.986508] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10757.360143] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10760.890860] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [10768.187213] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10774.739651] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10781.915526] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10782.204256] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10782.403354] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10788.369360] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [10796.025308] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10803.372301] Lustre: Permanently deactivating lustre-OST0000 [10803.398606] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [10811.939929] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [10812.966345] LustreError: 392934:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [10812.985322] LustreError: 392934:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [10812.996563] LustreError: 392934:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [10815.565197] LustreError: 392982:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [10815.573103] LustreError: 392982:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [10823.884081] Lustre: server umount lustre-OST0000 complete [10826.432922] LustreError: 393232:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [10826.437482] LustreError: 393232:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [10829.281129] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10829.291290] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10829.304545] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10829.310319] Lustre: Skipped 1 previous similar message [10831.841664] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10831.846967] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10836.961290] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10836.978941] Lustre: Skipped 2 previous similar messages [10839.083787] Lustre: server umount lustre-MDT0000 complete [10842.081196] LustreError: 389947:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10842.109262] LustreError: 389947:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10842.810779] LustreError: 390678:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788856022 with bad export cookie 1882548798007398074 [10842.816333] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10842.829962] LustreError: 390678:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10852.466466] Lustre: server umount lustre-MDT0001 complete [10860.869938] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10867.141407] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [10871.859813] LustreError: 394391:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [10882.746114] Lustre: 394732:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.104@tcp [10885.806601] Lustre: server umount MGS complete [10896.798894] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10897.128053] LustreError: 395456:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10897.157717] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [10897.165652] Lustre: Skipped 1 previous similar message [10897.244310] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10901.947991] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [10912.154678] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10912.559827] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [10917.208062] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [10925.386806] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10930.784337] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10932.548043] Lustre: Permanently reactivating lustre-OST0000 [10932.563738] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [10932.569343] Lustre: Skipped 2 previous similar messages [10962.400161] LustreError: 397287:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [10971.757895] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10972.065466] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10972.073376] Lustre: Skipped 1 previous similar message [10977.774536] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [10984.798501] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10991.586594] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10991.594337] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10991.616344] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10995.155139] Lustre: server umount lustre-OST0000 complete [10999.777187] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10999.785455] LustreError: Skipped 1 previous similar message [10999.791141] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10999.810202] Lustre: Skipped 1 previous similar message [10999.816168] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10999.826136] Lustre: Skipped 2 previous similar messages [11004.507847] Lustre: server umount lustre-MDT0000 complete [11007.977088] LustreError: 395457:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11007.997723] LustreError: 395457:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11008.027957] LustreError: 395437:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788856188 with bad export cookie 1882548798007399131 [11008.037097] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11008.039895] LustreError: 395437:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11008.535567] Lustre: server umount lustre-MDT0001 complete [11020.765885] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [11024.021541] Key type lgssc unregistered [11024.290183] LNet: 399917:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11024.296103] LNetError: 399917:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11025.327337] LNet: Removed LNI 192.168.203.104@tcp [11026.380708] Key type .llcrypt unregistered [11026.383676] Key type ._llcrypt unregistered [11044.379508] Key type ._llcrypt registered [11044.385899] Key type .llcrypt registered [11044.614570] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_hostid [11065.129450] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [11067.162467] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11067.250009] alg: No test for adler32 (adler32-zlib) [11068.523506] Lustre: Lustre: Build Version: 2.17.57_108_g4a8ba26 [11069.045722] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [11070.857049] Key type lgssc registered [11071.934425] Lustre: Echo OBD driver; http://www.lustre.org/ [11105.442181] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11105.483061] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11106.853741] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11106.900834] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11107.011282] Lustre: lustre-MDT0000: new disk, initializing [11107.107758] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11107.134717] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11111.823119] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [11121.519752] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11121.621507] Lustre: 403072:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11121.698822] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11121.702056] Lustre: Skipped 1 previous similar message [11121.764364] Lustre: lustre-MDT0001: new disk, initializing [11121.866535] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11121.912750] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11121.925442] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11126.392851] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [11135.692973] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11140.801690] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11146.843690] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11147.152329] Lustre: lustre-OST0000: new disk, initializing [11147.155094] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11147.164822] Lustre: 405040:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11147.265346] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11153.586450] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [11155.525243] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11155.534925] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11155.639510] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11163.291388] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11168.316225] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11168.513557] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11172.892469] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11173.127860] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11176.417927] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11176.419319] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11176.426602] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11176.469917] Lustre: Skipped 1 previous similar message [11181.559638] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11181.564817] Lustre: Skipped 2 previous similar messages [11186.670534] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11189.216820] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11189.422559] Lustre: server umount lustre-OST0000 complete [11193.826075] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11193.840718] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11193.868864] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11193.886455] Lustre: Skipped 1 previous similar message [11196.899752] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11202.019133] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11202.032739] Lustre: Skipped 3 previous similar messages [11207.136087] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11207.320360] Lustre: server umount lustre-MDT0000 complete [11210.874190] LustreError: 403062:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788856390 with bad export cookie 8730784459423585711 [11210.881135] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11210.887358] LustreError: 403062:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11211.347596] Lustre: server umount lustre-MDT0001 complete [11220.195374] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [11221.919630] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 04:33:20 (1788856400) [11229.487802] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11229.975281] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11235.308936] LustreError: 407505:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11235.324946] LustreError: 407505:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11235.507872] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [11239.394530] LustreError: 407504:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11244.514179] LustreError: 407505:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11246.372688] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11247.177266] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11252.412780] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [11263.804616] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11270.200815] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11279.022621] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11279.664829] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11287.374488] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [11297.869648] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11301.534052] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [11305.305753] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [11313.667349] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11313.682292] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11313.727331] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11313.754103] Lustre: Skipped 2 previous similar messages [11319.137852] Lustre: server umount lustre-OST0000 complete [11323.875047] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11323.884127] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11323.892086] Lustre: Skipped 1 previous similar message [11329.680974] Lustre: server umount lustre-MDT0000 complete [11330.018187] LustreError: 407501:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11330.033171] LustreError: 407501:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11333.569197] LustreError: 407485:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788856513 with bad export cookie 8730784459423586600 [11333.569905] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11333.580963] LustreError: 407485:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11334.106420] Lustre: server umount lustre-MDT0001 complete [11346.409241] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [11349.586389] Key type lgssc unregistered [11349.821607] LNet: 411234:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11349.829412] LNetError: 411234:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11349.843802] LNet: Removed LNI 192.168.203.104@tcp [11350.499222] Key type .llcrypt unregistered [11350.500556] Key type ._llcrypt unregistered [11373.887325] Key type ._llcrypt registered [11373.891354] Key type .llcrypt registered [11373.980019] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [11374.298831] Key type .llcrypt unregistered [11374.306051] Key type ._llcrypt unregistered [11394.543844] Key type ._llcrypt registered [11394.546696] Key type .llcrypt registered [11394.703413] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [11395.623770] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11395.685700] alg: No test for adler32 (adler32-zlib) [11396.749360] Lustre: Lustre: Build Version: 2.17.57_108_g4a8ba26 [11397.030886] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [11398.704159] Key type lgssc registered [11399.737817] Lustre: Echo OBD driver; http://www.lustre.org/ [11407.460955] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11407.477220] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11408.927539] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11413.695273] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [11418.084456] LustreError: 413439:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11418.130651] LustreError: 413439:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11421.870555] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11422.152841] LustreError: 413440:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11422.170620] LustreError: 413440:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11422.180589] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11427.304982] LustreError: 413439:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11427.324908] LustreError: 413439:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11428.549569] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [11432.425638] LustreError: 413440:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11432.447281] LustreError: 413440:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11433.461188] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [11435.187043] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11438.579825] LustreError: 413436:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11438.611988] LustreError: 413436:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11442.568215] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11443.225679] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11448.087473] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [11458.535467] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11458.546902] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11458.575565] Lustre: Skipped 1 previous similar message [11458.582685] LustreError: 414164:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11458.607752] LustreError: 414164:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11458.836936] Lustre: server umount lustre-OST0000 complete [11463.649875] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11463.653036] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11463.669692] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11463.691888] Lustre: Skipped 1 previous similar message [11465.231664] Lustre: server umount lustre-MDT0000 complete [11468.677700] LustreError: 413421:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788856648 with bad export cookie 16274474169445172596 [11468.697085] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11469.008892] Lustre: server umount lustre-MDT0001 complete [11480.171101] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [11482.902755] Key type lgssc unregistered [11483.178960] LNet: 416583:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11483.190508] LNetError: 416583:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11484.265853] LNet: Removed LNI 192.168.203.104@tcp [11485.058330] Key type .llcrypt unregistered [11485.064331] Key type ._llcrypt unregistered [11508.524093] Key type ._llcrypt registered [11508.525994] Key type .llcrypt registered [11508.679567] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [11509.942730] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11509.997401] alg: No test for adler32 (adler32-zlib) [11511.165341] Lustre: Lustre: Build Version: 2.17.57_108_g4a8ba26 [11511.554913] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [11513.344229] Key type lgssc registered [11515.103689] Lustre: Echo OBD driver; http://www.lustre.org/ [11525.260201] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11525.292307] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11541.024372] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [11546.144415] LustreError: 417974:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.104@tcp: failed processing log, type 4: rc = -110 [11572.835340] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11578.603095] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [11586.225928] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11592.844301] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11593.429887] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11593.524404] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [11598.055027] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [11598.822752] LustreError: 417999:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11598.843193] LustreError: 417999:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11603.946043] LustreError: 418000:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11607.032961] LustreError: 417999:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11607.074641] LustreError: 417999:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11607.627637] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11607.974829] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11614.633587] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [11624.084886] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11630.363888] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11638.757722] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11638.769297] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11638.774224] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11638.792642] Lustre: Skipped 1 previous similar message [11638.811224] Lustre: Skipped 1 previous similar message [11640.579483] Lustre: server umount lustre-OST0000 complete [11645.936112] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [11645.948313] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11645.969962] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11646.484957] Lustre: server umount lustre-MDT0000 complete [11649.003815] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11649.005241] LustreError: 419014:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11650.006710] LustreError: 417980:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788856830 with bad export cookie 14517860182057791414 [11650.014450] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11650.018468] LustreError: 417980:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11650.420788] Lustre: server umount lustre-MDT0001 complete [11665.798065] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [11669.184576] Key type lgssc unregistered [11669.436187] LNet: 421541:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11669.453229] LNetError: 421541:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11669.491217] LNet: Removed LNI 192.168.203.104@tcp [11670.507510] Key type .llcrypt unregistered [11670.509161] Key type ._llcrypt unregistered [11696.636451] Key type ._llcrypt registered [11696.642897] Key type .llcrypt registered [11696.805851] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [11697.730803] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11697.739816] alg: No test for adler32 (adler32-zlib) [11698.889412] Lustre: Lustre: Build Version: 2.17.57_108_g4a8ba26 [11699.218066] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [11700.872440] Key type lgssc registered [11701.836752] Lustre: Echo OBD driver; http://www.lustre.org/ [11711.544433] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11711.588581] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11713.270057] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11718.306560] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [11723.749356] LustreError: 422954:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11723.771852] LustreError: 422954:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11728.866179] LustreError: 422955:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11729.175535] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11729.546396] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11736.516749] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [11745.703663] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11752.841504] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11760.477942] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11760.848404] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11767.865222] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [11776.051037] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11783.411895] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [11793.690191] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.4@tcp (stopping) [11793.902592] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11793.930148] Lustre: Skipped 2 previous similar messages [11794.221576] Lustre: server umount lustre-MDT0000 complete [11797.578751] LustreError: 422934:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.4@tcp arrived at 1788856977 with bad export cookie 14975356661127233811 [11797.606191] LustreError: 422934:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11799.015452] LustreError: 422949:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11799.044884] LustreError: 422949:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11800.098700] LustreError: 425556:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788856980 with bad export cookie 14975356661127233146 [11800.117536] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11804.131958] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11804.138306] LustreError: 424349:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11804.141034] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11804.141169] Lustre: Skipped 3 previous similar messages [11804.166819] LustreError: 424349:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11805.160438] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11810.280362] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11810.284270] LustreError: 422954:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11810.310115] LustreError: 422954:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11814.369311] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11814.599792] Lustre: server umount lustre-OST0000 complete [11820.519701] LustreError: 422954:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11820.544461] LustreError: 422954:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11822.277141] Lustre: server umount lustre-MDT0001 complete [11834.580875] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [11838.535055] Key type lgssc unregistered [11838.844744] LNet: 426662:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11838.855649] LNetError: 426662:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11838.874689] LNet: Removed LNI 192.168.203.104@tcp [11839.619967] Key type .llcrypt unregistered [11839.621853] Key type ._llcrypt unregistered [11868.437242] Key type ._llcrypt registered [11868.440321] Key type .llcrypt registered [11868.606373] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [11869.883953] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11870.011456] alg: No test for adler32 (adler32-zlib) [11871.148671] Lustre: Lustre: Build Version: 2.17.57_108_g4a8ba26 [11871.514869] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [11873.184210] Key type lgssc registered [11874.659600] Lustre: Echo OBD driver; http://www.lustre.org/ [11883.569413] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11883.598160] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11885.271818] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11890.575415] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [11895.783654] LustreError: 428079:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11895.803589] LustreError: 428079:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11900.668397] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11900.901920] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [11901.169173] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11905.950273] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [11914.431295] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11920.899966] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11928.242219] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11928.635571] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11930.688760] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [11935.251920] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [11942.878632] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11949.798784] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.4@tcp (stopping) [11951.078532] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11951.085979] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11951.109437] Lustre: Skipped 1 previous similar message [11951.143772] Lustre: Skipped 1 previous similar message [11954.922572] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.4@tcp (stopping) [11960.037608] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.4@tcp (stopping) [11960.043520] Lustre: Skipped 2 previous similar messages [11962.336437] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11962.725177] Lustre: server umount lustre-MDT0001 complete [11966.443422] LustreError: 428079:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11966.482163] LustreError: 428079:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11971.556095] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11971.560027] LustreError: 428078:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11971.561059] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11971.610097] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11971.651781] Lustre: Skipped 2 previous similar messages [11971.922247] Lustre: server umount lustre-OST0000 complete [11976.678184] LustreError: 428079:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11977.529873] Lustre: server umount lustre-MDT0000 complete [11995.425750] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [11999.590117] Key type lgssc unregistered [11999.941992] LNet: 431765:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11999.948428] LNetError: 431765:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11999.994906] LNet: Removed LNI 192.168.203.104@tcp [12001.344279] Key type .llcrypt unregistered [12001.346583] Key type ._llcrypt unregistered [12032.551796] Key type ._llcrypt registered [12032.565876] Key type .llcrypt registered [12032.849375] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_hostid [12054.230669] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [12055.713001] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12055.736499] alg: No test for adler32 (adler32-zlib) [12056.769901] Lustre: Lustre: Build Version: 2.17.57_108_g4a8ba26 [12057.039107] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [12058.728683] Key type lgssc registered [12059.983760] Lustre: Echo OBD driver; http://www.lustre.org/ [12099.858662] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [12116.380794] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12116.410649] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12117.686521] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12117.741292] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12117.826586] Lustre: lustre-MDT0000: new disk, initializing [12117.929942] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12117.956142] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12122.862798] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [12138.778541] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12139.259491] Lustre: lustre-OST0000: new disk, initializing [12139.269429] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12139.278124] Lustre: Skipped 1 previous similar message [12139.282599] Lustre: 436798:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12139.346950] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12148.097902] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [12148.384488] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [12148.405186] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [12148.445191] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [12158.681478] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12175.388104] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12175.503832] Lustre: 435779:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12175.564739] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12175.643492] Lustre: lustre-MDT0001: new disk, initializing [12175.768901] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12175.814096] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [12175.824944] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [12180.120711] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [12190.145482] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [12190.384789] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [12195.504056] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [12195.837499] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [12203.490086] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12203.492411] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12203.493696] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12203.513339] Lustre: Skipped 1 previous similar message [12207.179740] Lustre: server umount lustre-MDT0001 complete [12213.731541] LustreError: 435787:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12213.748147] LustreError: 435787:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [12218.861016] LustreError: 437317:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12218.873535] LustreError: 437317:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12223.976091] LustreError: 435786:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12224.015586] LustreError: 435786:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12229.089640] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12229.091264] LustreError: 435791:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12229.110674] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12229.110681] Lustre: Skipped 2 previous similar messages [12234.210722] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12234.211686] LustreError: 438739:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12239.330925] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12240.352152] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12240.598808] Lustre: server umount lustre-OST0000 complete [12244.453127] LustreError: 438739:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12244.479956] LustreError: 438739:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12259.296124] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12259.728505] Lustre: server umount lustre-MDT0000 complete [12275.089254] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [12278.084855] Key type lgssc unregistered [12278.415649] LNet: 440552:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12278.420879] LNetError: 440552:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12278.452862] LNet: Removed LNI 192.168.203.104@tcp [12279.325980] Key type .llcrypt unregistered [12279.334589] Key type ._llcrypt unregistered [12316.555759] Key type ._llcrypt registered [12316.558744] Key type .llcrypt registered [12316.706430] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [12317.154272] Key type .llcrypt unregistered [12317.157926] Key type ._llcrypt unregistered [12333.813807] Key type ._llcrypt registered [12333.825062] Key type .llcrypt registered [12333.981947] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [12335.065819] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12335.210040] alg: No test for adler32 (adler32-zlib) [12336.499817] Lustre: Lustre: Build Version: 2.17.57_108_g4a8ba26 [12336.952898] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [12338.657295] Key type lgssc registered [12339.749333] Lustre: Echo OBD driver; http://www.lustre.org/ [12363.526979] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12363.543987] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12364.813211] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12364.852613] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12364.905898] Lustre: lustre-MDT0000: new disk, initializing [12364.972583] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12364.994470] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12367.708724] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [12376.054404] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12376.162455] Lustre: 443920:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12376.215762] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12376.221544] Lustre: Skipped 1 previous similar message [12376.281759] Lustre: lustre-MDT0001: new disk, initializing [12376.377278] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12376.418903] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12376.429247] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12382.021464] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [12392.240299] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12398.202431] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12403.631331] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12403.966194] Lustre: lustre-OST0000: new disk, initializing [12403.972332] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12403.978756] Lustre: 445886:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12404.116734] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12409.758648] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [12411.960156] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12411.971885] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12412.049446] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12418.452498] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12427.755906] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12427.756700] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12427.763727] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12427.782790] Lustre: Skipped 2 previous similar messages [12430.589525] Lustre: server umount lustre-MDT0000 complete [12434.350818] LustreError: 443912:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788857614 with bad export cookie 17293511504974221990 [12434.357061] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12434.361489] LustreError: 443912:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12434.808807] Lustre: server umount lustre-MDT0001 complete [12448.539771] Lustre: server umount lustre-OST0000 complete [12459.351172] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 04:53:57 (1788857637) [12466.356607] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12466.861936] LustreError: 447917:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12466.876202] LustreError: 447917:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [12466.972477] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12470.705554] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [12472.290789] LustreError: 447918:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12476.409495] LustreError: 447917:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12478.852503] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12479.099722] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12484.413692] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [12492.602898] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12499.138685] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12517.557638] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12517.810627] Lustre: Found index 0 for lustre-OST0000, updating log [12517.972172] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12523.501712] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [12529.840808] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12537.480035] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [12546.018721] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12546.032647] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12546.047871] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12546.050557] Lustre: Skipped 2 previous similar messages [12547.110200] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12547.120968] Lustre: Skipped 1 previous similar message [12552.210458] Lustre: server umount lustre-OST0000 complete [12568.552254] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12568.564514] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12568.566097] Lustre: Skipped 2 previous similar messages [12568.587292] Lustre: Skipped 2 previous similar messages [12573.666984] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12573.682940] Lustre: Skipped 1 previous similar message [12573.962074] Lustre: server umount lustre-MDT0000 complete [12574.693364] LustreError: 447912:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12574.702316] LustreError: 447912:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12577.526464] LustreError: 447899:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788857757 with bad export cookie 17293511504974223390 [12577.558550] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12577.884895] Lustre: server umount lustre-MDT0001 complete [12586.843580] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 04:56:05 (1788857765) [12586.848409] LNet: 1 local NIs in recovery (showing 1): 192.168.203.104@tcp [13202.658497] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13203.121517] LustreError: 454359:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13203.197256] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13207.519716] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [13208.546554] LustreError: 454360:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13213.672754] LustreError: 454359:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13215.746929] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13216.162643] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13219.890556] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [13225.809975] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13230.220836] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13248.820694] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13249.028058] Lustre: Found index 0 for lustre-OST0000, updating log [13249.307344] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13251.383373] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:65) [13255.286255] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [13262.329863] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13264.736428] LNet: 1 local NIs in recovery (showing 1): 192.168.203.104@tcp [13264.877573] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13264.912583] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13278.176146] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13278.588058] Lustre: server umount lustre-OST0000 complete [13284.648636] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13284.682884] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13284.702160] Lustre: Skipped 1 previous similar message [13286.754333] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13286.760549] Lustre: Skipped 2 previous similar messages [13290.067466] Lustre: server umount lustre-MDT0000 complete [13290.859807] LustreError: 454356:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13290.885795] LustreError: 454356:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13294.651677] LustreError: 454341:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788858474 with bad export cookie 17293511504974224475 [13294.666072] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13294.673069] LustreError: 454341:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [13294.984835] Lustre: server umount lustre-MDT0001 complete [13323.944865] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_hostid [13333.374203] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [13367.613301] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 05:09:06 (1788858546) [13385.581472] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [13388.808784] Key type lgssc unregistered [13389.166395] LNet: 460971:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13389.171870] LNetError: 460971:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13509.995351] LNet: Removed LNI 192.168.203.104@tcp [13511.019221] Key type .llcrypt unregistered [13511.025346] Key type ._llcrypt unregistered [13527.339712] Key type ._llcrypt registered [13527.343985] Key type .llcrypt registered [13527.436580] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [13528.827601] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13528.854629] alg: No test for adler32 (adler32-zlib) [13529.878480] Lustre: Lustre: Build Version: 2.17.57_108_g4a8ba26 [13530.073640] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [13531.712162] Key type lgssc registered [13532.843912] Lustre: Echo OBD driver; http://www.lustre.org/ [13537.186623] LNet: Added LNI 192.168.203.104@tcp100 [8/256/0/180] [13538.086270] LNet: Added LNI 192.168.203.104@tcp101 [8/256/0/180] [13538.953495] LNet: Added LNI 192.168.203.104@tcp102 [8/256/0/180] [13539.807694] LNet: Added LNI 192.168.203.104@tcp103 [8/256/0/180] [13540.653842] LNet: Added LNI 192.168.203.104@tcp104 [8/256/0/180] [13541.491978] LNet: Added LNI 192.168.203.104@tcp105 [8/256/0/180] [13542.433829] LNet: Added LNI 192.168.203.104@tcp106 [8/256/0/180] [13543.332605] LNet: Added LNI 192.168.203.104@tcp107 [8/256/0/180] [13544.196523] LNet: Added LNI 192.168.203.104@tcp108 [8/256/0/180] [13544.980863] LNet: Added LNI 192.168.203.104@tcp109 [8/256/0/180] [13545.915762] LNet: Added LNI 192.168.203.104@tcp110 [8/256/0/180] [13546.872854] LNet: Added LNI 192.168.203.104@tcp111 [8/256/0/180] [13547.660284] LNet: Added LNI 192.168.203.104@tcp112 [8/256/0/180] [13548.629483] LNet: Added LNI 192.168.203.104@tcp113 [8/256/0/180] [13549.401774] LNet: Added LNI 192.168.203.104@tcp114 [8/256/0/180] [13550.303244] LNet: Added LNI 192.168.203.104@tcp115 [8/256/0/180] [13551.310701] LNet: Added LNI 192.168.203.104@tcp116 [8/256/0/180] [13552.145665] LNet: Added LNI 192.168.203.104@tcp117 [8/256/0/180] [13553.288961] LNet: Added LNI 192.168.203.104@tcp118 [8/256/0/180] [13554.426902] LNet: Added LNI 192.168.203.104@tcp119 [8/256/0/180] [13555.517689] LNet: Added LNI 192.168.203.104@tcp120 [8/256/0/180] [13556.526813] LNet: Added LNI 192.168.203.104@tcp121 [8/256/0/180] [13557.574154] LNet: Added LNI 192.168.203.104@tcp122 [8/256/0/180] [13558.514854] LNet: Added LNI 192.168.203.104@tcp123 [8/256/0/180] [13559.414579] LNet: Added LNI 192.168.203.104@tcp124 [8/256/0/180] [13560.359836] LNet: Added LNI 192.168.203.104@tcp125 [8/256/0/180] [13561.120528] LNet: Added LNI 192.168.203.104@tcp126 [8/256/0/180] [13562.044937] LNet: Added LNI 192.168.203.104@tcp127 [8/256/0/180] [13562.990351] LNet: Added LNI 192.168.203.104@tcp128 [8/256/0/180] [13563.948818] LNet: Added LNI 192.168.203.104@tcp129 [8/256/0/180] [13564.807880] LNet: Added LNI 192.168.203.104@tcp130 [8/256/0/180] [13565.734291] LNet: Added LNI 192.168.203.104@tcp131 [8/256/0/180] [13566.787757] LNet: Added LNI 192.168.203.104@tcp132 [8/256/0/180] [13567.792103] LNet: Added LNI 192.168.203.104@tcp133 [8/256/0/180] [13568.784229] LNet: Added LNI 192.168.203.104@tcp134 [8/256/0/180] [13569.727019] LNet: Added LNI 192.168.203.104@tcp135 [8/256/0/180] [13576.696554] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13576.728908] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13577.963422] Lustre: 464002:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.104@tcp: can't add NID 192.168.203.104@tcp130: rc = -75 [13578.052255] LNetError: 464025:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.104 [13578.065678] LNetError: Protocol error connecting to 192.168.203.104@tcp117 at host 192.168.203.104:988: is it running a compatible version of Lustre? [13578.215572] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13578.247388] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13578.385280] Lustre: lustre-MDT0000: new disk, initializing [13578.472907] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13578.487576] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13579.043675] LNetError: 461460:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.104 [13579.065154] LNetError: Protocol error connecting to 192.168.203.104@tcp111 at host 192.168.203.104:988: is it running a compatible version of Lustre? [13583.512628] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [13593.296500] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13593.440429] Lustre: 464017:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13593.498150] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13593.502621] Lustre: Skipped 1 previous similar message [13593.584192] Lustre: lustre-MDT0001: new disk, initializing [13593.619602] Lustre: 464822:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.104@tcp: can't add NID 192.168.203.104@tcp130: rc = -75 [13593.627920] Lustre: 464822:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [13593.767232] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13593.845586] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13593.854692] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13598.968153] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [13599.856449] Lustre: 465119:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.104@tcp: can't add NID 192.168.203.104@tcp130: rc = -75 [13599.875776] Lustre: 465119:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [13609.216731] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13613.032703] LNetError: 461459:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.104 [13613.059632] LNetError: Protocol error connecting to 192.168.203.104@tcp111 at host 192.168.203.104:988: is it running a compatible version of Lustre? [13615.341599] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13617.127768] LNetError: 461457:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.104 [13617.152941] LNetError: Protocol error connecting to 192.168.203.104@tcp117 at host 192.168.203.104:988: is it running a compatible version of Lustre? [13622.381366] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13622.692458] Lustre: lustre-OST0000: new disk, initializing [13622.700405] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13622.711356] Lustre: 465997:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13622.824140] Lustre: 466002:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.104@tcp: can't add NID 192.168.203.104@tcp130: rc = -75 [13622.840955] Lustre: 466002:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [13622.899797] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13626.341417] LNetError: 464029:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.104 [13626.349526] LNetError: Protocol error connecting to 192.168.203.104@tcp111 at host 192.168.203.104:988: is it running a compatible version of Lustre? [13629.560423] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [13629.983929] Lustre: 466369:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.104@tcp: can't add NID 192.168.203.104@tcp130: rc = -75 [13629.995122] Lustre: 466369:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [13630.080817] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13630.090547] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13630.223789] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13638.296939] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13638.632499] LNetError: 464026:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.104 [13638.643902] LNetError: 464026:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 1 previous similar message [13638.656123] LNetError: Protocol error connecting to 192.168.203.104@tcp111 at host 192.168.203.104:988: is it running a compatible version of Lustre? [13638.671263] LNetError: Skipped 1 previous similar message [13645.794687] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13645.807327] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13645.822789] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13650.914900] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13650.923162] Lustre: Skipped 1 previous similar message [13651.491795] Lustre: server umount lustre-OST0000 complete [13655.010606] LNetError: 461460:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.104 [13655.011858] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13655.024364] LNetError: 461460:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 1 previous similar message [13655.036606] LustreError: Skipped 1 previous similar message [13655.036647] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13655.036651] Lustre: Skipped 1 previous similar message [13655.048319] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13655.052074] LNetError: Protocol error connecting to 192.168.203.104@tcp117 at host 192.168.203.104:988: is it running a compatible version of Lustre? [13655.052096] LNetError: Skipped 1 previous similar message [13655.118597] Lustre: Skipped 1 previous similar message [13656.037391] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13661.153845] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13661.165626] Lustre: Skipped 3 previous similar messages [13666.273974] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13666.288474] Lustre: Skipped 1 previous similar message [13669.345729] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13669.704466] Lustre: server umount lustre-MDT0000 complete [13673.314045] LustreError: 464012:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788858853 with bad export cookie 157540260342170801 [13673.328604] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13673.664585] Lustre: server umount lustre-MDT0001 complete [13684.647093] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [13687.929207] Key type lgssc unregistered [13688.259836] LNet: 467908:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13688.269433] LNetError: 467908:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13688.277616] LNet: Removed LNI 192.168.203.104@tcp [13689.312285] LNet: Removed LNI 192.168.203.104@tcp100 [13689.325171] LNetError: 467908:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13689.336673] LNetError: 467908:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [13689.353168] LNet: Removed LNI 192.168.203.104@tcp101 [13690.400324] LNet: Removed LNI 192.168.203.104@tcp102 [13690.411399] LNetError: 467908:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13690.434830] LNetError: 467908:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [13690.450713] LNet: Removed LNI 192.168.203.104@tcp103 [13691.488154] LNet: Removed LNI 192.168.203.104@tcp104 [13692.513936] LNet: Removed LNI 192.168.203.104@tcp105 [13692.520128] LNetError: 467908:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13692.543874] LNetError: 467908:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [13692.560277] LNet: Removed LNI 192.168.203.104@tcp106 [13693.600149] LNet: Removed LNI 192.168.203.104@tcp107 [13694.626313] LNet: Removed LNI 192.168.203.104@tcp108 [13695.648334] LNet: Removed LNI 192.168.203.104@tcp109 [13696.672192] LNet: Removed LNI 192.168.203.104@tcp110 [13696.680904] LNetError: 467908:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13696.693184] LNetError: 467908:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [13696.697635] LNet: Removed LNI 192.168.203.104@tcp111 [13697.760223] LNet: Removed LNI 192.168.203.104@tcp112 [13698.787276] LNet: Removed LNI 192.168.203.104@tcp113 [13699.809178] LNet: Removed LNI 192.168.203.104@tcp114 [13700.832188] LNet: Removed LNI 192.168.203.104@tcp115 [13701.856146] LNet: Removed LNI 192.168.203.104@tcp116 [13702.880234] LNet: Removed LNI 192.168.203.104@tcp117 [13702.884675] LNet: Removed LNI 192.168.203.104@tcp118 [13702.886312] LNet: Removed LNI 192.168.203.104@tcp119 [13702.888261] LNet: Removed LNI 192.168.203.104@tcp120 [13702.891334] LNet: Removed LNI 192.168.203.104@tcp121 [13702.896839] LNet: Removed LNI 192.168.203.104@tcp122 [13702.902976] LNet: Removed LNI 192.168.203.104@tcp123 [13702.908698] LNet: Removed LNI 192.168.203.104@tcp124 [13702.914540] LNet: Removed LNI 192.168.203.104@tcp125 [13702.918408] LNet: Removed LNI 192.168.203.104@tcp126 [13702.925645] LNet: Removed LNI 192.168.203.104@tcp127 [13702.928780] LNet: Removed LNI 192.168.203.104@tcp128 [13702.932118] LNet: Removed LNI 192.168.203.104@tcp129 [13702.940249] LNet: Removed LNI 192.168.203.104@tcp130 [13702.943095] LNet: Removed LNI 192.168.203.104@tcp131 [13702.951287] LNet: Removed LNI 192.168.203.104@tcp132 [13702.955771] LNet: Removed LNI 192.168.203.104@tcp133 [13702.961854] LNet: Removed LNI 192.168.203.104@tcp134 [13702.971206] LNet: Removed LNI 192.168.203.104@tcp135 [13703.560436] Key type .llcrypt unregistered [13703.562560] Key type ._llcrypt unregistered [13727.269385] Key type ._llcrypt registered [13727.272564] Key type .llcrypt registered [13727.366788] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_hostid [13743.263609] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [13744.646563] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13744.795172] alg: No test for adler32 (adler32-zlib) [13745.956636] Lustre: Lustre: Build Version: 2.17.57_108_g4a8ba26 [13746.321924] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [13748.112861] Key type lgssc registered [13749.463868] Lustre: Echo OBD driver; http://www.lustre.org/ [13784.175391] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 05:16:02 (1788858962) [13804.606210] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [13807.600791] Key type lgssc unregistered [13807.924514] LNet: 471990:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13807.934487] LNetError: 471990:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13807.961485] LNet: Removed LNI 192.168.203.104@tcp [13808.925250] Key type .llcrypt unregistered [13808.927555] Key type ._llcrypt unregistered [13826.422892] Key type ._llcrypt registered [13826.425965] Key type .llcrypt registered [13826.538516] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [13827.943043] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13828.157874] alg: No test for adler32 (adler32-zlib) [13829.301628] Lustre: Lustre: Build Version: 2.17.57_108_g4a8ba26 [13829.770672] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [13831.552187] Key type lgssc registered [13833.219125] Lustre: Echo OBD driver; http://www.lustre.org/ [13843.711409] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13843.746556] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13845.077262] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13845.119805] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13845.276336] Lustre: lustre-MDT0000: new disk, initializing [13845.346419] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13845.366301] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13850.332397] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [13860.092124] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13860.169434] Lustre: 473202:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13860.222053] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13860.225202] Lustre: Skipped 1 previous similar message [13860.271133] Lustre: lustre-MDT0001: new disk, initializing [13860.325619] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13860.366974] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13860.376351] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13864.898735] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [13876.563095] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13885.139673] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13895.937032] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13896.326820] Lustre: lustre-OST0000: new disk, initializing [13896.330968] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13896.339316] Lustre: 475172:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13896.412925] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13902.993762] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [13904.515147] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13904.538836] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13904.833375] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13914.237249] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13917.990843] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing lnet_if_list [13924.312846] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [13924.347082] LNet: Added LNI 192.168.203.104@tcp10 [8/256/0/180] [13930.630217] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [13930.662373] LNet: Added LNI 192.168.203.104@tcp11 [8/256/0/180] [13934.874393] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [13934.897683] LNet: Added LNI 192.168.203.104@tcp12 [8/256/0/180] [13939.900688] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [13939.925834] LNet: Added LNI 192.168.203.104@tcp13 [8/256/0/180] [13944.348577] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [13944.432515] LNet: Added LNI 192.168.203.104@tcp14 [8/256/0/180] [13949.526856] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [13949.547342] LNet: Added LNI 192.168.203.104@tcp15 [8/256/0/180] [13954.997620] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [13955.035707] LNet: Added LNI 192.168.203.104@tcp16 [8/256/0/180] [13962.043568] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [13962.127485] LNet: Added LNI 192.168.203.104@tcp17 [8/256/0/180] [13970.214695] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [13970.284775] LNet: Added LNI 192.168.203.104@tcp18 [8/256/0/180] [13976.021818] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [13976.067373] LNet: Added LNI 192.168.203.104@tcp19 [8/256/0/180] [13981.952937] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [13981.971443] LNet: Added LNI 192.168.203.104@tcp20 [8/256/0/180] [13987.492099] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [13987.532903] LNet: Added LNI 192.168.203.104@tcp21 [8/256/0/180] [13994.303900] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [13994.343385] LNet: Added LNI 192.168.203.104@tcp22 [8/256/0/180] [14000.815636] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [14000.850974] LNet: Added LNI 192.168.203.104@tcp23 [8/256/0/180] [14007.068908] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [14007.097901] LNet: Added LNI 192.168.203.104@tcp24 [8/256/0/180] [14012.817502] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [14012.855195] LNet: Added LNI 192.168.203.104@tcp25 [8/256/0/180] [14018.890665] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [14018.915599] LNet: Added LNI 192.168.203.104@tcp26 [8/256/0/180] [14025.290520] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [14025.318704] LNet: Added LNI 192.168.203.104@tcp27 [8/256/0/180] [14032.048246] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [14032.090044] LNet: Added LNI 192.168.203.104@tcp28 [8/256/0/180] [14038.638406] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [14038.677533] LNet: Added LNI 192.168.203.104@tcp29 [8/256/0/180] [14045.577708] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [14045.617620] LNet: Added LNI 192.168.203.104@tcp30 [8/256/0/180] [14052.344535] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [14052.394211] LNet: Added LNI 192.168.203.104@tcp31 [8/256/0/180] [14059.279471] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [14059.321161] LNet: Added LNI 192.168.203.104@tcp32 [8/256/0/180] [14066.436293] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [14066.458898] LNet: Added LNI 192.168.203.104@tcp33 [8/256/0/180] [14074.486519] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [14074.508377] LNet: Added LNI 192.168.203.104@tcp34 [8/256/0/180] [14081.515669] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [14081.553815] LNet: Added LNI 192.168.203.104@tcp35 [8/256/0/180] [14088.982749] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [14089.023718] LNet: Added LNI 192.168.203.104@tcp36 [8/256/0/180] [14096.510286] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [14096.540633] LNet: Added LNI 192.168.203.104@tcp37 [8/256/0/180] [14104.608846] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [14104.645213] LNet: Added LNI 192.168.203.104@tcp38 [8/256/0/180] [14113.247536] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [14113.295246] LNet: Added LNI 192.168.203.104@tcp39 [8/256/0/180] [14122.970806] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [14123.025132] LNet: Added LNI 192.168.203.104@tcp40 [8/256/0/180] [14132.559179] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [14132.588107] LNet: Added LNI 192.168.203.104@tcp41 [8/256/0/180] [14139.387864] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [14139.407467] LNet: Added LNI 192.168.203.104@tcp42 [8/256/0/180] [14146.021970] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [14146.056328] LNet: Added LNI 192.168.203.104@tcp43 [8/256/0/180] [14153.831210] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [14153.862792] LNet: Added LNI 192.168.203.104@tcp44 [8/256/0/180] [14162.945599] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [14162.965780] LNet: Added LNI 192.168.203.104@tcp45 [8/256/0/180] [14171.740938] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [14171.783862] LNet: Added LNI 192.168.203.104@tcp46 [8/256/0/180] [14180.257455] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [14180.309798] LNet: Added LNI 192.168.203.104@tcp47 [8/256/0/180] [14187.999062] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [14188.023314] LNet: Added LNI 192.168.203.104@tcp48 [8/256/0/180] [14195.699441] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [14195.721177] LNet: Added LNI 192.168.203.104@tcp49 [8/256/0/180] [14203.258920] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [14203.291179] LNet: Added LNI 192.168.203.104@tcp50 [8/256/0/180] [14222.308313] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14222.311546] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14222.336098] Lustre: Skipped 2 previous similar messages [14222.347289] Lustre: Skipped 2 previous similar messages [14227.428158] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14227.435253] Lustre: Skipped 2 previous similar messages [14227.725472] Lustre: server umount lustre-MDT0000 complete [14231.318183] LustreError: 473999:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788859411 with bad export cookie 11435360933859134233 [14231.320766] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14231.334927] LustreError: 473999:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [14231.774174] Lustre: server umount lustre-MDT0001 complete [14248.931647] Lustre: 472472:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788859412/real 1788859412] req@00000000adf1e541 x1875754624394624/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788859428 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14248.955042] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14249.847774] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [14252.064412] Lustre: 472472:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788859416/real 1788859416] req@000000000b2c03dd x1875754624394752/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788859432 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14257.120437] Lustre: 472474:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788859421/real 1788859421] req@00000000093ad07d x1875754624395136/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788859437 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14257.154644] Lustre: 472474:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [14257.929158] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14262.240730] Lustre: 472471:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788859426/real 1788859426] req@0000000060d47e62 x1875754624395264/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788859442 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14276.646871] LustreError: 472470:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000576d5089 x1875754624396032/t0(0) o250->MGC192.168.203.104@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14277.137583] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14278.193852] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14283.233871] LustreError: 475166:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14283.251391] LustreError: 475166:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [14283.932168] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [14288.365977] LustreError: 475166:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14292.477740] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14292.918402] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14293.966424] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14298.326073] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [14307.180598] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14314.613351] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14340.076954] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14340.107511] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14340.126315] Lustre: Skipped 1 previous similar message [14345.192169] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14345.202122] Lustre: Skipped 1 previous similar message [14346.004498] Lustre: server umount lustre-OST0000 complete [14350.305547] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14350.320241] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14350.335757] Lustre: Skipped 2 previous similar messages [14356.272516] Lustre: server umount lustre-MDT0000 complete [14360.366765] LustreError: 473191:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788859540 with bad export cookie 11435360933859136837 [14360.369252] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14360.383404] LustreError: 473191:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [14360.921791] Lustre: server umount lustre-MDT0001 complete [14372.907415] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [14376.156328] Key type lgssc unregistered [14376.686711] LNet: 502217:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14376.694334] LNetError: 502217:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14376.701582] LNet: Removed LNI 192.168.203.104@tcp [14376.702977] LNet: Removed LNI 192.168.203.104@tcp10 [14376.704502] LNet: Removed LNI 192.168.203.104@tcp11 [14376.705841] LNet: Removed LNI 192.168.203.104@tcp12 [14376.708545] LNet: Removed LNI 192.168.203.104@tcp13 [14376.710793] LNet: Removed LNI 192.168.203.104@tcp14 [14376.712623] LNet: Removed LNI 192.168.203.104@tcp15 [14376.714840] LNet: Removed LNI 192.168.203.104@tcp16 [14376.717164] LNet: Removed LNI 192.168.203.104@tcp17 [14376.718326] LNet: Removed LNI 192.168.203.104@tcp18 [14376.719624] LNet: Removed LNI 192.168.203.104@tcp19 [14376.721166] LNet: Removed LNI 192.168.203.104@tcp20 [14376.722636] LNet: Removed LNI 192.168.203.104@tcp21 [14376.724486] LNet: Removed LNI 192.168.203.104@tcp22 [14376.726210] LNet: Removed LNI 192.168.203.104@tcp23 [14376.728276] LNet: Removed LNI 192.168.203.104@tcp24 [14376.730027] LNet: Removed LNI 192.168.203.104@tcp25 [14376.733133] LNet: Removed LNI 192.168.203.104@tcp26 [14376.734600] LNet: Removed LNI 192.168.203.104@tcp27 [14376.735892] LNet: Removed LNI 192.168.203.104@tcp28 [14376.737233] LNet: Removed LNI 192.168.203.104@tcp29 [14376.738862] LNet: Removed LNI 192.168.203.104@tcp30 [14376.740570] LNet: Removed LNI 192.168.203.104@tcp31 [14376.743321] LNet: Removed LNI 192.168.203.104@tcp32 [14376.745428] LNet: Removed LNI 192.168.203.104@tcp33 [14376.747623] LNet: Removed LNI 192.168.203.104@tcp34 [14376.749847] LNet: Removed LNI 192.168.203.104@tcp35 [14376.753486] LNet: Removed LNI 192.168.203.104@tcp36 [14376.754930] LNet: Removed LNI 192.168.203.104@tcp37 [14376.760287] LNet: Removed LNI 192.168.203.104@tcp38 [14376.764842] LNet: Removed LNI 192.168.203.104@tcp39 [14376.769066] LNet: Removed LNI 192.168.203.104@tcp40 [14376.774901] LNet: Removed LNI 192.168.203.104@tcp41 [14376.781652] LNet: Removed LNI 192.168.203.104@tcp42 [14376.786212] LNet: Removed LNI 192.168.203.104@tcp43 [14376.789888] LNet: Removed LNI 192.168.203.104@tcp44 [14376.798460] LNet: Removed LNI 192.168.203.104@tcp45 [14376.803609] LNet: Removed LNI 192.168.203.104@tcp46 [14376.813163] LNet: Removed LNI 192.168.203.104@tcp47 [14376.817655] LNet: Removed LNI 192.168.203.104@tcp48 [14376.826275] LNet: Removed LNI 192.168.203.104@tcp49 [14376.840504] LNet: Removed LNI 192.168.203.104@tcp50 [14377.942930] Key type .llcrypt unregistered [14377.947155] Key type ._llcrypt unregistered [14406.851807] Key type ._llcrypt registered [14406.854354] Key type .llcrypt registered [14407.022659] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_hostid [14422.120082] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [14423.679255] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14423.706900] alg: No test for adler32 (adler32-zlib) [14424.800693] Lustre: Lustre: Build Version: 2.17.57_108_g4a8ba26 [14425.144168] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [14426.840175] Key type lgssc registered [14428.279410] Lustre: Echo OBD driver; http://www.lustre.org/ [14461.551335] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14461.575180] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14462.815933] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14462.880068] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14463.002374] Lustre: lustre-MDT0000: new disk, initializing [14463.098410] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14463.113823] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14467.142931] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [14477.831297] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14478.036076] Lustre: 505414:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14478.161352] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14478.165614] Lustre: Skipped 1 previous similar message [14478.242401] Lustre: lustre-MDT0001: new disk, initializing [14478.333521] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14478.364289] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14478.376709] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14482.751248] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [14491.780989] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14497.931695] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14506.924460] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14507.229777] Lustre: *** cfs_fail_loc=90f, val=0*** [14507.333853] Lustre: lustre-OST0000: new disk, initializing [14507.337635] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14507.342224] Lustre: 507431:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14507.439658] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14515.031639] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [14516.800654] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14516.818561] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14517.043374] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14524.567963] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14532.585410] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14532.589811] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14532.596269] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14532.604170] Lustre: Skipped 1 previous similar message [14532.609624] LustreError: Skipped 1 previous similar message [14537.699482] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14537.708223] Lustre: Skipped 2 previous similar messages [14542.836842] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14542.840884] Lustre: Skipped 1 previous similar message [14546.403360] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14546.829041] Lustre: server umount lustre-OST0000 complete [14553.059113] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14553.073867] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14553.082481] Lustre: Skipped 1 previous similar message [14558.094980] Lustre: server umount lustre-MDT0000 complete [14562.033341] LustreError: 505405:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788859742 with bad export cookie 16809266255581114553 [14562.034529] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14562.041496] LustreError: 505405:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [14563.058785] Lustre: server umount lustre-MDT0001 complete [14574.354253] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [14578.131894] Key type lgssc unregistered [14578.497817] LNet: 509336:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14578.525085] LNetError: 509336:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14578.563970] LNet: Removed LNI 192.168.203.104@tcp [14579.661513] Key type .llcrypt unregistered [14579.663921] Key type ._llcrypt unregistered [14599.875159] Key type ._llcrypt registered [14599.877024] Key type .llcrypt registered [14600.008659] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [14601.176525] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14601.242869] alg: No test for adler32 (adler32-zlib) [14602.312074] Lustre: Lustre: Build Version: 2.17.57_108_g4a8ba26 [14602.532388] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [14604.208584] Key type lgssc registered [14605.433744] Lustre: Echo OBD driver; http://www.lustre.org/ [14613.359633] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14613.397816] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14614.887901] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14619.341874] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [14624.238297] LustreError: 510603:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14624.274386] LustreError: 510603:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [14627.991340] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14628.397921] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14633.133942] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [14640.805783] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14646.883293] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14654.008993] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14654.532359] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14661.430514] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [14669.135590] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14683.546898] loop0: detected capacity change from 0 to 204800000 [14683.763807] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14683.797708] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14683.845749] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14691.700271] loop0: detected capacity change from 0 to 204800000 [14692.009381] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14692.029673] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14692.044736] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14698.040843] loop0: detected capacity change from 0 to 204800000 [14704.541645] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14704.701413] Lustre: fs15246-OST0000: new disk, initializing [14704.708314] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [14704.724243] Lustre: 514199:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14704.807630] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14711.380775] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [14721.025978] loop1: detected capacity change from 0 to 204800000 [14726.573891] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14726.749513] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [14726.790931] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [14726.878097] Lustre: fs15246-MDT0000: new disk, initializing [14727.174065] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14727.221458] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14727.381800] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [14728.417978] Lustre: 515321:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@000000000967579c x1875755434818688/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788859924 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [14732.681837] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [14737.508269] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [14737.527388] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [14737.579462] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [14755.304658] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14755.319327] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [14760.140289] Lustre: server umount fs15246-MDT0000 complete [14760.424788] LustreError: 510598:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14765.541674] LustreError: 512445:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14768.116711] Lustre: server umount fs15246-OST0000 complete [14780.904155] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14780.930438] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14780.939330] Lustre: Skipped 1 previous similar message [14786.033973] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14786.040373] Lustre: Skipped 1 previous similar message [14791.138385] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14791.155468] Lustre: Skipped 1 previous similar message [14794.208106] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14794.370496] Lustre: server umount lustre-OST0000 complete [14801.391328] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14801.394787] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14801.409334] Lustre: Skipped 2 previous similar messages [14801.426441] Lustre: Skipped 1 previous similar message [14803.806172] Lustre: server umount lustre-MDT0000 complete [14806.502242] LustreError: 510600:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14806.522944] LustreError: 510600:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [14806.906509] LustreError: 510585:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788859986 with bad export cookie 17046225490241279380 [14806.919397] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14806.923176] LustreError: 510585:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [14807.208733] Lustre: server umount lustre-MDT0001 complete [14816.472850] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [14819.075498] Key type lgssc unregistered [14819.351480] LNet: 518418:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14819.359193] LNetError: 518418:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14819.382879] LNet: Removed LNI 192.168.203.104@tcp [14820.212659] Key type .llcrypt unregistered [14820.215231] Key type ._llcrypt unregistered [14838.135512] Key type ._llcrypt registered [14838.139484] Key type .llcrypt registered [14838.246790] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [14839.492322] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14839.608066] alg: No test for adler32 (adler32-zlib) [14840.693523] Lustre: Lustre: Build Version: 2.17.57_108_g4a8ba26 [14840.983725] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [14842.664177] Key type lgssc registered [14843.880033] Lustre: Echo OBD driver; http://www.lustre.org/ [14881.069889] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_hostid [14890.560660] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [14925.462677] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14925.496187] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14926.788406] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14926.839433] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14926.904248] Lustre: lustre-MDT0000: new disk, initializing [14927.016544] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14927.031159] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14931.143670] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [14940.924037] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14941.009398] Lustre: 522881:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14941.060234] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14941.068968] Lustre: Skipped 1 previous similar message [14941.163242] Lustre: lustre-MDT0001: new disk, initializing [14941.252168] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14941.311924] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14941.331504] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14945.960761] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [14955.660450] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14961.497987] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14968.278394] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14968.685545] Lustre: lustre-OST0000: new disk, initializing [14968.700706] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14968.707757] Lustre: 524848:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14968.820981] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14975.935361] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [14976.602773] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14976.629110] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14976.766173] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14986.339380] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14992.284895] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [14992.600800] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [14998.528437] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [14998.789493] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [15002.602308] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15002.613787] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15002.637132] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15002.637904] LustreError: Skipped 1 previous similar message [15002.655937] Lustre: Skipped 1 previous similar message [15007.716750] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15007.726424] Lustre: Skipped 1 previous similar message [15012.851147] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15015.397257] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15015.752500] Lustre: server umount lustre-OST0000 complete [15023.079889] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15023.084334] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15023.100712] Lustre: Skipped 1 previous similar message [15023.104817] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15023.110230] Lustre: Skipped 2 previous similar messages [15028.193920] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15028.202110] Lustre: Skipped 1 previous similar message [15034.337579] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15034.495898] Lustre: server umount lustre-MDT0000 complete [15038.877810] LustreError: 522873:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788860218 with bad export cookie 959052639988062230 [15038.889096] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15038.900836] LustreError: 522873:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15039.563874] Lustre: server umount lustre-MDT0001 complete [15067.719476] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_hostid [15078.606278] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [15110.911557] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 14844 sec ========= 05:38:08 (1788860288) [15113.023312] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 05:38:11 (1788860291) === [15116.974630] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 05:38:15 (1788860295) === [15159.147876] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [15161.676102] Key type lgssc unregistered [15162.051046] LNet: 531502:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15162.057871] LNetError: 531502:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15163.110678] LNet: Removed LNI 192.168.203.104@tcp [15164.047788] Key type .llcrypt unregistered [15164.050299] Key type ._llcrypt unregistered